greyhaven-ai/autocontext

What it solves

It addresses the difficulty of consistently improving AI agent performance on specific tasks. Instead of manually tweaking prompts, it provides a recursive harness that automates the process of testing, evaluating, and distilling lessons from agent attempts to ensure future iterations succeed.

How it works

The system runs a specified goal against an evaluation framework. It iteratively executes the task, keeping successful strategies and discarding failures. This process generates a filesystem-first trail of evidence, including traces, reports, playbooks, and datasets. These artifacts can then be used to guide the agent in subsequent runs or even be used to distill stable behaviors into cheaper local-model training artifacts.

Who it’s for

Developers and AI researchers building agents who want to move beyond trial-and-error prompting toward a systematic, evidence-driven approach to agent optimization and reliability.

Highlights

• Recursive Improvement: Automatically iterates on tasks to refine agent behavior.
• Multi-Provider Support: Integrates with various runtimes including Pi, Anthropic, OpenAI-compatible APIs, and Claude Code via MCP.
• Knowledge Persistence: Creates permanent playbooks, hints, and toolsets for specific scenarios.
• Distillation: Includes a train command to distill successful behaviors into local models.
• Filesystem-First: All runs leave behind transparent, inspectable artifacts like JSONL traces and Markdown reports.