Anthropic Harness Design for Long-Running Application Development

TL;DR

Anthropic released a three‑agent harness (planner, generator, evaluator) that lets Claude autonomously create rich full‑stack applications over multi‑hour runs, addressing both aesthetic judgment in frontend design and reliable code quality in long‑running coding tasks.

Why naive long‑running agents fail

Naïve implementations of autonomous coding agents quickly lose coherence as the context window fills, leading to "context anxiety" where the model wraps up work prematurely. Simple context compaction does not reset the model’s internal state, so anxiety persists. Additionally, agents that self‑evaluate tend to over‑rate their own outputs, especially for subjective tasks like design, resulting in bland or buggy artifacts.

GAN‑inspired multi‑agent architecture

Anthropic adapted the generator‑evaluator pattern from Generative Adversarial Networks:

  • Generator creates code or UI assets.
  • Evaluator grades the output against concrete criteria and provides detailed feedback.
  • Planner (added for full‑stack work) expands a short user prompt into a detailed product spec, setting the scope for downstream agents.

Separating generation from evaluation allows the evaluator to be tuned for skepticism, giving the generator concrete signals for improvement.

Frontend design: turning taste into gradable criteria

To make subjective aesthetics measurable, Anthropic defined four grading criteria:

  1. Design quality – coherence of colors, typography, layout, and imagery.
  2. Originality – presence of custom decisions versus stock templates.
  3. Craft – technical execution such as spacing, contrast, and hierarchy.
  4. Functionality – usability independent of visual appeal.

The evaluator was calibrated with few‑shot examples and used Playwright to interact with live pages before scoring. Iterative loops (5‑15 cycles per generation) drove the generator toward more distinctive designs, with notable creative leaps such as a 3‑D CSS museum site emerging after ten iterations.

Scaling to full‑stack development

Architecture

  • Planner – converts a 1‑4 sentence prompt into a 16‑feature spec across ten sprints, adding AI‑enhanced features where possible.
  • Generator – implements one feature per sprint using a React/Vite/FastAPI/SQLite (later PostgreSQL) stack, commits to git, and self‑evaluates before handoff.
  • Evaluator – uses Playwright to run end‑to‑end tests, grading each sprint on product depth, visual design, functionality, and code quality. A sprint contract is negotiated before implementation to align expectations.

Execution details

  • Context resets, required for Claude Sonnet 4.5 due to context anxiety, were dropped when switching to Claude Opus 4.5, whose longer context handling eliminated the need for frequent resets.
  • Communication between agents occurs via structured files, preserving state across sessions without over‑specifying implementation details.

Comparative results: harness vs. solo runs

Retro video‑game maker

  • Solo run – produced a basic UI but suffered from layout waste, rigid workflow, and a non‑functional game core.
  • Harness run – generated a richer spec, more polished UI, integrated Claude‑driven AI features, and a playable game (albeit with minor physics glitches). The evaluator caught 27 sprint‑level criteria, preventing many bugs that appeared in the solo version.
  • Cost – the harness was >20× more expensive, but the quality gap was immediately evident.

Digital Audio Workstation (DAW) prototype

  • Run duration – ~4 hours, $124 token cost.
  • Outcome – functional arrangement view, mixer, and transport; AI‑assisted composition demonstrated.
  • Evaluator findings – missing core interactions (drag‑and‑drop clips, instrument panels, visual effect editors) and stubbed audio recording, highlighting remaining gaps even with Opus 4.5.

Harness simplification after Opus 4.6

Opus 4.6 improved long‑context retrieval, planning, and self‑debugging, prompting a systematic reduction of harness components:

  • Removed sprint decomposition – the model handled longer coherent runs without explicit chunking.
  • Evaluator moved to end‑of‑run pass – still valuable for edge‑case verification, but unnecessary for tasks comfortably within the model’s solo capability.
  • Planner retained – essential for expanding terse prompts into actionable specs.

The simplification showed that evaluator utility is workload‑dependent: it adds cost when the task exceeds the model’s baseline ability, but can be omitted when the model alone suffices.

Lessons and future directions

  1. Iterate on the target model – Always test the harness with the current model version; capabilities can render parts of the scaffold obsolete.
  2. Decompose when needed – Task chunking and specialized agents provide headroom for complex problems, but should be stripped once the model can handle the workload natively.
  3. Separate generation from evaluation – An external, skeptical evaluator yields more tractable tuning than trying to make the generator self‑critical.
  4. Concrete grading criteria turn subjectivity into feedback – Encoding design principles as gradable terms drives aesthetic improvement.
  5. Harness design remains a moving target – As models improve, new combinations of agents will emerge, keeping the space of useful harnesses vibrant.

Appendix

Example planner output for a DAW prompt (omitted for brevity).

Sources