davebcn87/pi-autoresearch

What it solves

pi-autoresearch provides an autonomous optimization loop for the pi AI coding agent. It allows an agent to iteratively try an idea, benchmark the result, keep improvements, and revert regressions without human intervention, automating the tedious process of performance tuning or metric optimization.

How it works

The project separates infrastructure (the extension) from domain knowledge (the skill). The agent uses a set of tools to initialize a session, run benchmark commands, and log results. All session state—including objectives, metrics, and a history of every run—is persisted in a .auto/ directory containing a prompt file and a JSONL log. This allows the agent to resume work across restarts or context resets. To ensure quality, it supports optional backpressure checks (tests/linting) and uses Median Absolute Deviation (MAD) to calculate a confidence score, helping the agent distinguish real gains from benchmark noise.

Who it’s for

Developers using the pi AI coding agent who want to automate the optimization of specific targets such as test runtime, bundle size, LLM training loss, build speeds, or Lighthouse performance scores.

Highlights

• Autonomous Loop: Fully automated edit-commit-benchmark-revert cycle.
• Persistence: Session state is saved to disk, allowing agents to rehydrate context after conversation compaction.
• Confidence Scoring: Uses statistical noise estimation to flag whether improvements are likely real or marginal.
• Reviewable Output: A finalization skill groups successful experiments into clean, independent git branches for human review.
• Extensibility: Supports pre- and post-iteration hooks for tasks like external research or notifications.