davebcn87/pi-autoresearch

pi‑autoresearch – autonomous optimisation loops for the pi coding agent

What it is – An extension (plus a few “skills”) for the open‑source terminal‑based AI coding assistant pi (https://pi.dev). It gives pi the ability to run autoresearch cycles: propose a change, benchmark it, decide whether to keep or revert, and repeat automatically until a stopping condition is met.

Why it matters – The core idea of autoresearch (originally demonstrated by Andrej Karpathy) is to let an LLM act as a self‑improving engineer. pi‑autoresearch packages that workflow into a reusable, domain‑agnostic toolchain, so you can optimise anything that can be measured with a command line metric – test speed, bundle size, build time, Lighthouse scores, even LLM training loss.

Key components

Component	Role
Extension (installed via `pi install npm:pi-autoresearch`)	Provides three low‑level tools (`init_experiment`, `run_experiment`, `log_experiment`) and a live dashboard widget that lives above the editor.
Skill `autoresearch-create`	Interactive wizard that asks (or infers) the optimisation goal, creates a git branch, writes the session files (`.auto/prompt.md`, `.auto/measure.sh`, etc.) and starts the loop.
Skill `autoresearch-finalize`	After the loop, reads the log, groups successful runs into independent, reviewable git branches.
Skill `autoresearch-hooks` (optional)	Lets you drop `before.sh` / `after.sh` scripts that run around each iteration, receiving JSON context and feeding back free‑form messages to the agent.

How a session works

Create – Run /skill:autoresearch-create. Pi asks for a goal (e.g. “reduce test runtime”), a command to measure it, and the metric (seconds, KB, score, etc.). It writes:
- .auto/prompt.md – a living description of the experiment.
- .auto/measure.sh – a script that runs the command and prints METRIC name=number lines.
- (optional) .auto/checks.sh – correctness checks that run after each successful benchmark.
Loop – The agent repeatedly:
- edits code,
- run_experiment → executes measure.sh and records wall‑clock time,
- log_experiment → appends a JSON line to .auto/log.jsonl, auto‑commits, updates the widget.
- decides to keep (leave the commit) or revert (reset to previous state) based on the metric and optional confidence scoring.
- The loop runs unattended until you interrupt it or a maxIterations limit is hit.
Monitor – Use the built‑in widget, /autoresearch dashboard (full‑screen terminal view) or /autoresearch export (live browser dashboard) to watch a table of runs, a confidence score, and a spinner for the current experiment.
Finalize – When satisfied, run /skill:autoresearch-finalize. Pi groups the kept runs into separate branches, each containing a single logical change, ready for code‑review.

Confidence scoring (optional)

After three runs the extension computes a confidence value = |best_improvement| / MAD (median absolute deviation) of the metric values. It is shown as green (≥ 2×, likely real), yellow (1‑2×, marginal) or red (< 1×, within noise). The score is advisory only – the agent still decides whether to keep a change.

Extensibility

Domain‑agnostic core – The extension knows only how to run a command, log the result, and manage git. All domain knowledge lives in the skill you write (or use the provided autoresearch-create wizard).
Hooks – Drop scripts in .auto/hooks/ to fetch external data, send desktop notifications, rotate ideas, etc. Hooks receive a JSON payload on stdin and can return a short message that pi will treat as a steering cue.
Configuration – .auto/config.json can set workingDir (override where files are read/written) and maxIterations (hard cap on experiments).

Installation & quick start

# from a pi session
pi install npm:pi-autoresearch   # pulls the extension + skills
# then start a new loop
/skill:autoresearch-create

You can also install manually by copying the extensions/ and skills/ folders into ~/.pi/agent/ and reloading pi.

Safety & cost notes

The extension runs arbitrary shell scripts (measure.sh, checks.sh, hooks) and makes git commits – run it in a dedicated branch or worktree.
Because the loop can invoke an LLM on every iteration, token usage can grow quickly. Control spend via your provider’s API‑key limits or by setting maxIterations.

License

MIT – free to use, modify, and redistribute.