neosigmaai/auto-harness

What it solves

auto-harness is a framework for the automated self-improvement of AI coding agents. It solves the problem of manually iterating on system prompts and tool definitions to improve an agent's performance on specific benchmarks. Instead of a human developer iterating, a coding agent is tasked with optimizing another agent's code and prompts, creating a self-improving loop.

How it works

The system operates as a closed-loop optimization process:

1. Execution: The system runs a benchmark (e.g., Tau-bench, Terminal-Bench 2.0, or BIRD-Interact) and records the results.
2. Analysis: A coding agent analyzes the failure traces from the training set to diagnose why the agent failed.
3. Improvement: The coding agent edits the target agent's file (agent/agent.py) to improve its system prompt or tools.
4. Gating: Every change is passed through a three-step gate: a regression suite of previously passed tasks, a full test set evaluation to ensure the mean reward is improved, and a suite promotion step where new passing tasks are added to the regression suite.
5. Recording: Results are appended to a history log and the agent records its findings in a learnings.md file.
6. Repeat: The process repeats, allowing the agent to iterate overnight.

Who it’s for

Developers and AI researchers who want to automate the agentic own-improvement loop for LLM-based agents, particularly those focusing on tool-use, SQL generation, or terminal-based tasks.

Highlights

• Benchmark-Agnostic: Works with any benchmark that provides per-task rewards.
• Self-Maintained Evals: The coding agent automatically manages the suite.json regression suite without manual curation.
• Structural Anti-Cheating: Prevents data leakage by ensuring the coding agent only has access to training traces, not test traces.
• Integrated Benchmarks: Comes with built-in support for Tau-bench, Terminal-Bench 2.0, and BIRD-Interact.