RyanAlberts/best-of-Agent-Harnesses

What it is

best-of-Agent-Harnesses is a curated, ranked list of ~166 open-source projects centered on one idea: an AI "agent harness" — the runtime machinery that turns a language model that answers into an agent that acts.

The README's crisp definition (credited to Simon Willison): "an LLM agent runs tools in a loop to achieve a goal." The harness is everything around that loop: what tools exist, what needs approval, what the model sees each turn, and what survives a crash. In the framing of Andrej Karpathy, the model is "the kernel process of a new Operating System" and the harness is the rest of that OS — its scheduler, permissions, and memory.

Why it claims to matter

Because agents act, they fail: harness quality — retry logic, sandboxing, guardrails — determines whether agents actually ship. The repo cites benchmark data to back this up, e.g. swapping the harness changed pass rates more than many model upgrades did (same model going from 23% to 52% pass@1 on one model, 15% to 36% on another; and a bare model at ~30% on ARC-AGI-3 vs 95.5% in the right harness). It also notes harness rankings barely transfer across models (rank correlation −0.05), so the right harness for one model may be wrong for another — which is why the repo ships a query tool to help match harness to model (see below).

What's in the repo

• The list itself. Projects are ranked by relevance to harness concerns (environment, orchestration, lifecycle, guardrails) and by GitHub stars/activity, with tag chips (like mcp, memory) for cross-filtering. There are two visual maps of the whole landscape: one plotting projects by "adoption surface area" (simplicity↔capability) against stars, and one by autonomy (how much unsupervised rope) vs recovery (what happens when a run dies). In the tables, ★ marks headless-ready projects, ✱ marks durable ones.
• Decision guides ("comparisons" pages) for common choices: e.g. terminal coding agents, multi-agent orchestration frameworks, memory layers, sandboxing, browser agents, eval platforms — plus higher-level essays like Why the harness matters more than the model and How to pick a harness.
• Templates & playbooks. Copy-paste setup files (e.g. one AGENTS.md that works across 7 coding tools, safe Claude Code permission settings, a ~180-line minimal agent harness in Python) and step-by-step "build your own harness in an hour" walkthroughs.
• Skills (10 installable). Small scripts your own agent can load that run on your machine and report back auditable results: test-drive coding agents on tasks from your own repos, check which instruction files your agents actually load, verify guardrails against ~90 dangerous commands, detect false "tests pass" claims, find where tokens and money are wasted, etc. Anything that changes files or spends money waits for your explicit yes.
• A searchable website (one page per harness, filter by capability, autonomy, recovery).
• Machine-readable access for agents themselves: an MCP server (with commands like recommend and pick_harness), plus llms.txt and harnesses.json, so an agent can query the list to recommend a harness matched to its own model and task — instead of inheriting whatever harness someone else benchmarked. (An MCP server is, roughly, a standard interface that lets AI agents call external tools and data.)

The README ends with a "Pick by use case" reader's index — dozens of categories like "turnkey coding agent today," "always-on personal agent that lives in my chat apps," "memory layer," "multi-agent orchestration," "browser agents," "sandboxed code execution," "evaluate/benchmark agents" — each pointing to a shortlist and the relevant table section.

Caveats

Contributions are welcome (per the badge), and the whole thing acknowledges its own framing by pointing to a separate page (Who says so, what they measured, and what the claim does not mean). The list is opinionated by design: it invests in harness quality rather than framework plumbing, citing guidance like Anthropic's "build simple" advice.