Anthropic Harness Design for Long-Running Application Development

TL;DR

Anthropic released a three‑agent harness (planner, generator, evaluator) that lets Claude autonomously produce rich front‑end designs and full‑stack applications over multi‑hour runs, overcoming context‑window limits and self‑evaluation bias.

Why naive long‑running agents fail

  • Context anxiety – As Claude’s context window fills, the model prematurely wraps up work or loses coherence. Simple compaction keeps history but does not reset the model’s internal state, so Claude still behaves anxiously. Full context resets, paired with structured hand‑off artifacts, give the model a clean slate and prevent premature termination.
  • Self‑evaluation bias – When an agent grades its own output, it tends to over‑praise, especially on subjective tasks like design. Separating generation from evaluation allows a dedicated evaluator to be tuned toward skepticism, providing concrete feedback that the generator can iterate on.

"Agents tend to respond by confidently praising their own work—even when a human observer sees it as mediocre." – Anthropic engineering post

Turning subjective design quality into gradable criteria

Anthropic defined four explicit grading criteria for front‑end design:

  1. Design quality – Cohesion of colors, typography, layout, imagery, and mood.
  2. Originality – Presence of custom decisions versus stock or generic AI patterns.
  3. Craft – Technical execution such as spacing, contrast ratios, and hierarchy.
  4. Functionality – Usability independent of aesthetics.

Weighting emphasized design quality and originality because Claude already performed well on craft and functionality. The evaluator was calibrated with few‑shot examples to align scores with the author’s preferences.

The generator created HTML/CSS/JS pages, while the evaluator used Playwright to interact with the live page, screenshot, and produce a detailed critique. Iterations (5‑15 per run) alternated between generation and evaluation, with the generator deciding whether to refine the current direction or pivot entirely.

Observed outcomes

  • Scores improved across iterations, though not always linearly.
  • The wording of criteria directly shaped the visual style (e.g., “museum quality” steered designs toward a particular aesthetic).
  • A notable creative leap occurred at iteration 10 of a Dutch art‑museum site, where the generator switched from a conventional landing page to a 3D CSS‑based spatial experience.

Extending the GAN‑inspired pattern to full‑stack development

Three‑agent architecture

Agent Role
Planner Expands a 1‑4 sentence prompt into a detailed product spec, identifies AI‑enhancement opportunities, and avoids over‑specifying low‑level details.
Generator Implements the spec feature‑by‑feature (sprints) using a React‑Vite‑FastAPI‑SQLite/PostgreSQL stack, maintains git history, and self‑evaluates before hand‑off.
Evaluator Executes the running app via Playwright, checks UI, API, and database behavior, and grades each sprint against criteria (design, functionality, depth, code quality).

Key mechanisms

  • Sprint contracts – Before each sprint, generator proposes deliverables and verification steps; evaluator reviews and approves, ensuring alignment with the high‑level spec.
  • File‑based hand‑offs – Agents communicate by reading/writing structured files, preserving state without requiring a persistent conversation.
  • Automatic context compaction – Using Claude Agent SDK, the harness lets Opus 4.5 handle growing context without explicit resets, thanks to the model’s reduced “context anxiety.”

Comparative benchmark: Solo vs. full harness

Harness Duration Cost
Solo Claude run (single agent) 20 min $9
Full three‑agent harness 6 hr $200

The harness produced a far more feature‑rich retro game maker, with a polished UI, AI‑assisted sprite generation, and a functional play mode, whereas the solo run suffered from layout waste, broken entity wiring, and non‑functional gameplay.

Simplifying the harness after Opus 4.6

  • Opus 4.6 improved long‑context handling, planning, and self‑debugging, prompting a removal of the sprint decomposition.
  • The planner and evaluator remained valuable; without the planner, the generator under‑scoped the project.
  • Evaluator was shifted to a single end‑of‑run QA pass. On tasks within Opus 4.6’s solo capability, the evaluator added little value; on edge‑case tasks it still caught critical bugs.

Updated cost breakdown (DAW generation)

Phase Duration Cost
Planner 4.7 min $0.46
Build Round 1 2 hr 7 min $71.08
QA Round 1 8.8 min $3.24
Build Round 2 1 hr 2 min $36.89
QA Round 2 6.8 min $3.09
Build Round 3 10.9 min $5.88
QA Round 3 9.6 min $4.06
Total 3 hr 50 min $124.70

The evaluator still identified missing core DAW interactions (e.g., draggable clips, instrument panels), demonstrating that even with a stronger model, targeted QA remains a high‑leverage component.

Lessons and future directions

  1. Context resets vs. compaction – Resetting the model’s context eliminates “context anxiety” but adds orchestration overhead. When a model (e.g., Opus 4.5) exhibits strong anxiety, resets are essential; newer models may rely solely on compaction.
  2. Separate evaluator – Tuning a dedicated evaluator to be skeptical is more tractable than making the generator self‑critical. The evaluator’s feedback loop drives iterative improvement.
  3. Gradable subjective criteria – Encoding design principles into explicit scores converts vague aesthetic judgments into actionable feedback.
  4. Iterative harness pruning – As models improve, systematically remove components that no longer provide measurable lift. The evaluator’s utility is task‑dependent, not binary.
  5. Hybrid decomposition – High‑level planning plus feature‑level sprint contracts balance scope control with flexibility, preventing early spec errors from cascading.

Anthropic’s work shows that thoughtful harness design can extend the practical capabilities of current LLMs, and that the space of useful harness combinations expands—not contracts—as models become more capable.

Acknowledgements

Thanks to Mike Krieger, Michael Agaby, Justin Young, Jeremy Hadfield, David Hershey, Julius Tarng, Xiaoyi Zhang, Barry Zhang, Orowa Sidker, Michael Tingley, Ibrahim Madha, Martina Long, Canyon Robbins, Jake Eaton, Alyssa Leonard, and Stef Sequeira for their contributions.

Appendix: Sample planner output (RetroForge game maker)

RetroForge - 2D Retro Game Maker

Overview
RetroForge is a web‑based creative studio for designing and building 2D retro‑style video games. It combines the nostalgic charm of classic 8‑bit and 16‑bit game aesthetics with modern, intuitive editing tools—enabling anyone from hobbyist creators to indie developers to bring their game ideas to life without writing traditional code.

Features
1. Project Dashboard & Management
   * Create, open, duplicate, delete projects
   * Visual cards with thumbnails and timestamps
   * Metadata: name, description, resolution, tile size, palette
2. Tile‑based Level Editor
   * Drag‑and‑drop tiles, layers, scrolling viewport
3. Pixel‑art Sprite Editor
   * Palette picker, zoom, frame animation
4. Entity Behavior System
   * Visual scripting for movement, collisions
5. Playable Test Mode
   * Real‑time preview, keyboard controls
6. AI‑assisted tools
   * Prompt‑driven sprite generation, level layout suggestions

Sources

Related