Anthropic Demystifying evals for AI agents – Best practices and roadmap

TL;DR

Anthropic released a detailed framework for designing automated evaluations of AI agents, showing why rigorous evals are essential for reliable shipping and providing a practical roadmap—from task collection to long‑term maintenance.

What is an evaluation?

An evaluation (eval) is a test that feeds an input to an AI system and then applies grading logic to its output. Anthropic focuses on automated evals that run without human users during development.

  • Single‑turn evals: one prompt, one response, simple grader.
  • Multi‑turn evals: involve tool calls, state changes, and a sequence of reasoning steps.
  • Agent evals: combine multi‑turn interaction, environment mutation, and often creative model behavior that can out‑perform static expectations.

Core terminology

Term Meaning
Task / problem A single test case with defined inputs and success criteria.
Trial One attempt at a task; multiple trials capture stochasticity.
Grader Logic that scores an aspect of performance; may contain multiple checks.
Transcript / trace Full record of a trial, including all messages, tool calls, and intermediate results.
Outcome The final state of the environment (e.g., a reservation exists in a database).
Evaluation harness Infrastructure that runs tasks, records steps, grades outputs, and aggregates results.
Agent harness / scaffold System that enables a model to act as an agent (orchestrates tool calls).
Evaluation suite A collection of tasks targeting a specific capability (e.g., customer‑support scenarios).

Why evals matter

  • Early‑stage agents can be tested manually, but once in production teams lose visibility into regressions.
  • Without evals, debugging is reactive: wait for user complaints, reproduce manually, fix, and hope nothing else broke.
  • Evals provide baseline metrics, regression guards, and a communication channel between product and research.
  • Companies with eval pipelines can adopt new models in days rather than weeks, because they can quickly measure strengths, tune prompts, and verify safety.

Types of graders

  1. Code‑based graders – deterministic checks (e.g., unit tests, static analysis). Ideal for coding agents.
  2. Model‑based graders – LLMs acting as judges, using structured rubrics. Useful for subjective dimensions like tone or reasoning quality.
  3. Human graders – occasional calibration of model judges or verification of edge cases.

Grader scores can be combined weighted, binary, or hybrid depending on the task.

Capability vs. regression evals

  • Capability evals target low pass rates to surface weaknesses and guide improvement.
  • Regression evals aim for near‑100 % pass rates to catch unintended breakages.
  • Over time, capability suites can graduate to regression suites once the agent reliably solves the tasks.

Evaluation techniques by agent type

Coding agents

  • Use well‑specified tasks and stable test environments.
  • Deterministic graders (unit tests) are primary; e.g., SWE‑bench Verified and Terminal‑Bench.
  • Supplement with transcript graders for tool‑use patterns and code‑quality heuristics.

Conversational agents

  • Measure both outcome (e.g., ticket resolved) and interaction quality (tone, turn count).
  • Often require a second LLM to simulate the user.
  • Benchmarks such as τ‑Bench and τ2‑Bench evaluate multi‑turn, adversarial dialogues.

Research agents

  • Combine groundedness checks, coverage checks, and source‑quality checks.
  • Use LLM rubrics calibrated against expert humans because “correctness” is often subjective.
  • Example benchmark: BrowseComp (find needles in haystacks on the open web).

Computer‑use agents

  • Run agents in sandboxed GUIs (browser, OS) and verify final state (files, DB entries, UI elements).
  • Benchmarks: WebArena (browser tasks) and OSWorld (full OS control).
  • Evaluate tool‑selection efficiency: DOM extraction vs. screenshot‑based interaction.

Handling non‑determinism

  • pass@k – probability that at least one of k trials succeeds. Useful when a single success suffices.
  • pass^k – probability that all k trials succeed. Critical for consistency‑focused products.
  • Both metrics should be reported; choice depends on whether occasional success or guaranteed reliability is required.

Roadmap: From zero to trustworthy evals

Step Action Key Insight
0 Start early with 20‑50 real‑world failure cases. Small samples give high signal when agents change frequently.
1 Convert existing manual checks or bug‑tracker items into tasks. Ensures evals reflect actual user impact.
2 Write unambiguous task specs and reference solutions. Two experts should agree on pass/fail; ambiguous specs cause noisy metrics.
3 Balance positive and negative cases to avoid one‑sided optimization. Prevents over‑triggering or under‑triggering behaviors (e.g., unnecessary searches).
4 Build a stable, isolated eval harness (clean env per trial). Removes infrastructure noise and prevents unfair advantage from prior state.
5 Choose deterministic graders first, add LLM graders where needed, and calibrate with occasional human review. Rigid step‑by‑step grading penalizes creative solutions; grade outcomes instead of paths.
6 Regularly inspect transcripts to verify grader fairness and detect hidden bugs. Transcript review is the primary way to confirm that evals measure the intended behavior.
7 Watch for eval saturation (scores near 100 %). When saturation occurs, introduce harder tasks or new dimensions.
8 Maintain suites via dedicated eval teams plus contributions from product owners; treat suites like unit‑test libraries.

Integrating evals with other quality signals

  • Automated evals – run in CI/CD for every code/model change.
  • Production monitoring – detect distribution drift and real‑world failures.
  • A/B testing – validate significant changes once traffic volume permits.
  • User feedback & transcript sampling – continuously surface novel failure modes.
  • Systematic human studies – calibrate LLM judges and evaluate highly subjective outputs.

Together these layers form a Swiss‑cheese model of safety: each catches issues the others might miss.

Open‑source and commercial eval frameworks

Framework Strengths
Harbor Containerized agent execution, benchmark registry (e.g., Terminal‑Bench 2.0).
Braintrust Combines offline evals with production observability; includes autoevals library.
LangSmith / Langfuse Tight LangChain integration, tracing, dataset management; Langfuse is self‑hosted.
Arize Phoenix / AX Open‑source tracing and debugging; AX adds SaaS scaling and monitoring.

Choose a framework that fits your stack, then focus effort on building high‑quality tasks and graders.

Takeaway

Rigorous, automated evaluations are the backbone of reliable AI‑agent development. By defining clear tasks, using deterministic and calibrated graders, measuring both capability and regression, and maintaining suites over time, teams can ship agents faster, adopt new models confidently, and avoid costly reactive debugging cycles.

Sources