Browse Source

docs: add autoresearch case study to loop engineering research

- Maps autoresearch to the 5 building blocks
- Compares autoresearch vs Ralph vs Full Loop Engineering
- Identifies minimum viable loop: skill + git + evaluation
Michael Henke 1 month ago
parent
commit
4f4978d468
1 changed files with 55 additions and 1 deletions
  1. 55 1
      docs/loop-engineering-research.md

+ 55 - 1
docs/loop-engineering-research.md

@@ -240,6 +240,58 @@ The missing piece is the outer loop itself: the scheduler that runs on a timer,
 
 ---
 
+## Case Study: Andrej Karpathy's autoresearch
+
+[autoresearch](https://github.com/karpathy/autoresearch) (March 2026) is the purest real-world implementation of loop engineering. It's a minimalist autonomous AI research framework: one Python file to edit (`train.py`), one fixed evaluation harness (`prepare.py`), and one Markdown "skill" file (`program.md`) that drives an AI agent to run an infinite experiment loop.
+
+### How It Works
+
+1. The agent reads `program.md` which defines a `LOOP FOREVER` construct
+2. It modifies `train.py` with an experimental idea
+3. Runs a 5-minute training experiment on a single GPU
+4. Evaluates validation bits-per-byte (BPB)
+5. If improved: keeps the commit, advances the branch
+6. If same or worse: git resets to previous state
+7. Repeats indefinitely (~12 experiments/hour, ~100 per night)
+
+### Mapping to the 5 Building Blocks
+
+| Building Block | autoresearch | Notes |
+|---|---|---|
+| **Automations** | The agent IS the scheduler | `LOOP FOREVER` in program.md, no external cron |
+| **Worktrees** | Not used | Single-agent, single-file system |
+| **Skills** | `program.md` IS the skill | Explicitly called "a super lightweight skill" |
+| **Connectors** | Not used | Agent's native tools only (git, python) |
+| **Sub-agents** | Not used | Single-agent system |
+
+### Comparison to Ralph and Full Loop Engineering
+
+| Aspect | autoresearch | Ralph | Full Loop Engineering |
+|---|---|---|---|
+| Agent lifetime | Continuous (one session) | Fresh per iteration | Continuous with sub-agents |
+| Context | Grows within session | Reset each iteration | Persistent + fresh sub-agent contexts |
+| Memory | Git + results.tsv | Git + progress.txt + prd.json | Durable state files + skills |
+| Verification | Automated (val_bpb check) | Manual (human reviews diffs) | Maker/checker split |
+| Scheduling | Agent self-schedules | Manual trigger | Cron/events |
+| Parallelism | None | None | Parallel sub-agents |
+| Complexity | ~100 lines (program.md) | ~50 lines (Bash loop) | Hundreds of lines across 5 blocks |
+
+### Key Lessons
+
+1. **The skill file is the most important piece.** `program.md` defines the loop, success criteria, and failure handling. Without it, the agent has no idea what to do.
+
+2. **Git is sufficient memory for many loops.** No need for complex state management. The commit history IS the state.
+
+3. **A fixed time budget makes experiments comparable.** Every experiment runs for exactly 5 minutes, regardless of what the agent changes. This is critical for fair evaluation.
+
+4. **Fast-fail prevents wasted compute.** NaN or exploding loss (>100) causes immediate exit. No point running 5 minutes of garbage.
+
+5. **The agent can be its own scheduler.** For simple loops, no external cron is needed. The agent follows the loop instructions and keeps going.
+
+6. **Minimum viable loop = skill + git + evaluation.** You don't need the full 5-block stack to get value from loop engineering. Start simple, add complexity when the task demands it.
+
+---
+
 ## Key Takeaways
 
 1. **The shift is real.** The people building the most-used coding agents have stopped prompting by hand. The leverage moved from the model to the loop around it.
@@ -254,6 +306,8 @@ The missing piece is the outer loop itself: the scheduler that runs on a timer,
 
 6. **OpenCode is 60% there.** Skills, connectors, and sub-agents are mature. The loop engine and automations are designed but unimplemented. The sub-agent infrastructure is the strongest piece of the stack.
 
+7. **autoresearch proves the minimum viable loop.** You don't need the full 5-block stack. A skill file + git + evaluation metric is enough to run autonomous experiments overnight. Start simple, add complexity when the task demands it.
+
 ---
 
-*Sources: Addy Osmani (addyosmani.com), Peter Steinberger (@steipete), Boris Cherny (Anthropic), Matt Van Horn (techtwitter.com), Geoffrey Huntley (ghuntley.com/ralph), snarktank/ralph (GitHub), awesomeclaude.ai, lushbinary.com, StationX, The Register, The New Stack, Business Insider, and 100+ Reddit/X/YouTube/TikTok threads.*
+*Sources: Addy Osmani (addyosmani.com), Peter Steinberger (@steipete), Boris Cherny (Anthropic), Matt Van Horn (techtwitter.com), Geoffrey Huntley (ghuntley.com/ralph), snarktank/ralph (GitHub), Andrej Karpathy (autoresearch), awesomeclaude.ai, lushbinary.com, StationX, The Register, The New Stack, Business Insider, and 100+ Reddit/X/YouTube/TikTok threads.*