drivelineresearch/autoresearch-claude-code

What it is

autoresearch‑claude‑code is a Python‑based experiment framework that lets you run auto‑research loops with Anthropic’s Claude Code or OpenAI’s Codex agents. You give the agent a concrete goal (e.g., “reduce test‑suite runtime”), a fixed benchmark, and a list of source files it may edit. The framework creates a temporary Git branch, runs the agent turn‑by‑turn, measures the impact of each change, and automatically commits only those edits that show a statistically‑significant improvement while staying within a user‑defined budget (max runs, time, or compute). All state is stored in JSONL files and can be inspected or rolled back.

Who it’s for

• Researchers or engineers who want to evaluate how well code‑generation agents can optimise existing code bases.
• Teams that need a reproducible, budget‑constrained “self‑improving” loop without building a custom server or MLOps pipeline.
• Anyone interested in measuring the real‑world impact of Claude Code or Codex on a concrete software task.

Core concepts

Concept	What it does
Goal definition	A short natural‑language command (e.g., optimize test suite runtime) tells the agent what to optimise.
Benchmark	A deterministic scorer script (e.g., a test runner) that returns a numeric metric.
Budget	Limits on number of turns, total wall‑clock time, and optionally on Codex invocations.
Noise handling	The framework calibrates measurement noise on a baseline run and only accepts changes that improve the metric beyond that noise floor.
Atomic state	All experiment metadata is written atomically to JSONL; concurrent writes are locked to avoid corruption.
Pause / recovery	A sentinel file (.autoresearch‑off) can pause the loop; the system can resume from the last valid checkpoint.
Locked scorer	Files listed with --protect are hashed each turn; the agent cannot modify them, preventing accidental corruption of the evaluation script.

Installation (quick)

# clone the repo
gh repo clone drivelineresearch/autoresearch-claude-code
cd autoresearch-claude-code

# install Claude skill (default) or Codex skill
./install.sh --claude   # installs Claude hooks under ~/.agents/skills/autoresearch
# or
./install.sh --codex    # installs Codex skill
# or install both
./install.sh --all

The script creates symlinks; keep the checkout around so the links stay valid. Uninstall with the matching ./uninstall.sh flags.

Running a loop (Claude example)

1. Initialize – in the target project, run the skill command:

   $ autoresearch optimize test suite runtime with at most 20 runs
   

   The agent creates an autoresearch worktree, records a baseline, and pauses waiting for you to review.
2. Review – check status or report to see the baseline metric.
3. Resume – start the automated loop:

   python3 ~/.agents/skills/autoresearch/scripts/claude_loop.py \
     --max-turns 20 --max-seconds 3600 --protect path/to/scorer.py
   

   Adjust --protect for any files the scorer depends on.
4. Monitor – the loop prints turn summaries; you can abort by creating .autoresearch‑off.
5. Result – after the budget is exhausted, autoresearch report shows which commits were kept and the final metric.

The same workflow works for Codex, using the codex_loop.py script and the autoresearch CLI inside the Codex IDE.

Example use‑case

The repository ships an OpenBiomechanics demo that predicts baseball fastball velocity. It shows:

• How to define a custom scorer that outputs METRIC name=R2.
• How to plug in optional back‑ends (torch, boost, tabpfn, etc.) via uv sync --extra ….
• A reproducible run (./autoresearch.sh 42) that logs the entire search history.

Development & testing

• Core unit tests require only the Python standard library: python -m unittest discover -s tests -v.
• Example tests need scientific packages; they can be run after uv sync --project examples.
• Shell scripts are linted with shellcheck.
• CI runs on Linux/macOS and validates both the core and example suites.

Limitations & caveats

• Noise model is simple; it only checks that improvement exceeds the measured baseline variance. It does not provide statistical significance testing.
• The framework does not sandbox the agent; it relies on the “locked scorer” hashes and Git‑branch isolation to keep the original code safe.
• Budget is counted in turns and seconds, not in token usage or monetary cost.
• The OpenBiomechanics example’s historic R² jump (0.44 → 0.78) used data leakage; the repo notes that this is not a validated claim.
• Only Claude Code and Codex are supported; adding a new agent requires writing a skill wrapper and hook scripts.

License

MIT – you can freely use, modify, and redistribute the code.

---

Bottom line: autoresearch-claude-code gives you a ready‑made, reproducible loop to let Claude Code or Codex iteratively improve a codebase under strict budget and safety constraints, with built‑in logging, pause/resume, and baseline‑noise handling.