Anthropic: Demystifying Evals for AI Agents

TL;DR

Anthropic has released a detailed guide on building rigorous evaluations ("evals") for AI agents to prevent reactive debugging loops and accelerate development. The core takeaway is that effective agent evals require a combination of deterministic, model-based, and human graders, a distinction between capability and regression suites, and a commitment to reading transcripts to ensure graders are measuring actual performance rather than penalizing creativity.

The Architecture of Agent Evaluations

Evaluating AI agents is significantly more complex than evaluating single-turn LLM responses because agents operate over multiple turns, modify environment states, and can find creative solutions that may bypass static grading logic.

Key Evaluation Definitions

To build a consistent evaluation system, Anthropic defines the following components:

Task (Problem/Test Case): A single test with defined inputs and success criteria.
Trial: A single attempt at a task; multiple trials are run to account for model non-determinism.
Grader: Logic that scores performance via assertions or checks.
Transcript (Trace/Trajectory): The complete record of a trial, including tool calls, reasoning, and API interactions.
Outcome: The final state of the environment (e.g., whether a database record was actually created).
Evaluation Harness: The infrastructure that runs tasks concurrently, records steps, and aggregates results.
Agent Harness (Scaffold): The system enabling the model to act as an agent (e.g., Claude Code).
Evaluation Suite: A collection of tasks measuring specific capabilities or behaviors.

Evaluation Strategies by Agent Type

Different agent architectures require tailored grading techniques to ensure accuracy and utility.

Coding Agents

Coding agents are best evaluated using deterministic graders. Because software has binary outcomes (it works or it doesn't), the gold standard is running the generated code against unit tests in a stable environment.

Benchmarks: SWE-bench Verified and Terminal-Bench are cited as primary examples, where success is defined by fixing failing tests without breaking existing ones.
Hybrid Approach: While outcomes are deterministic, transcripts can be graded using LLM rubrics to assess code quality and tool-use efficiency.

Conversational Agents

Conversational agents require a multidimensional approach because the quality of the interaction is as important as the task completion.

Simulation: These evals often use a second LLM to simulate a user persona to stress-test the agent.
Grading: Success is measured by combining state checks (e.g., "Is the ticket resolved?"), transcript constraints (e.g., "Did it finish in <10 turns?"), and LLM rubrics for tone and empathy.

Research Agents

Research agents produce open-ended outputs where "correctness" is context-dependent.

Verification: Evals focus on groundedness (claims supported by sources), coverage (inclusion of key facts), and source quality.
Calibration: Because research quality is subjective, LLM-based rubrics must be frequently calibrated against expert human judgment.

Computer Use Agents

These agents interact with GUIs via screenshots and clicks rather than APIs.

Environment: Evaluation requires sandboxed environments (e.g., WebArena or OSWorld) where the final state of the OS or browser can be inspected.
Efficiency: Evals should track the agent's ability to choose the right tool (e.g., DOM extraction vs. screenshots) to balance latency and token cost.

Designing Effective Graders

Anthropic recommends a tiered approach to grading to balance speed, cost, and nuance.

Grader Type	Strengths	Weaknesses
Code-based	Fast, objective, reproducible, cheap.	Brittle to valid variations; lacks nuance.
Model-based	Flexible, scalable, handles nuance and open-ended tasks.	Non-deterministic; requires human calibration.
Human	Gold standard; matches expert judgment.	Expensive, slow, does not scale.

Capability vs. Regression Evals

Capability (Quality) Evals: Designed to find the ceiling of the agent's ability. They start with low pass rates to provide a "hill to climb."
Regression Evals: Designed to ensure existing functionality doesn't break. These should maintain a nearly 100% pass rate.
Lifecycle: Once a capability eval reaches a high pass rate, it "graduates" into the regression suite.

Handling Non-Determinism

Because agent behavior varies between runs, a single pass/fail is insufficient. Anthropic suggests two primary metrics:

pass@k: The probability that the agent gets at least one correct solution in k attempts. This is useful for tools where any single success is a win.
pass^k: The probability that all k trials succeed. This is critical for customer-facing agents where reliability and consistency are paramount.

Roadmap for Implementation

Phase 1: Dataset Collection

Start Small: Begin with 20-50 tasks derived from real failures.
Avoid Ambiguity: Ensure two experts would reach the same verdict. If an agent fails 100% of the time across many trials (0% pass@100), it usually indicates a broken task spec, not a failing model.
Balanced Sets: Include "negative" cases (when the agent should not perform an action) to prevent over-triggering behaviors.

Phase 2: Harness and Grader Design

Isolation: Ensure each trial starts from a clean environment to prevent shared state from artificially inflating scores.
Outcome over Path: Grade what the agent produced, not the specific sequence of tool calls it took, to avoid punishing creative but valid solutions.
Partial Credit: Implement scoring for tasks with multiple components to represent the continuum of success.

Phase 3: Long-term Maintenance

Transcript Review: Regularly read transcripts to distinguish between genuine agent mistakes and grading bugs.
Saturation Monitoring: When an eval reaches 100%, it no longer signals improvement. Teams must develop new, harder evals to continue measuring progress.
Eval-Driven Development: Define the eval tasks for a planned capability before building the feature, then iterate until the agent passes.

Holistic Performance Understanding

Automated evals are the first line of defense, but they must be part of a broader strategy:

Production Monitoring: Catches real-world distribution drift and unanticipated failures.
A/B Testing: Validates actual user outcomes (retention, completion) at scale.
User Feedback: Surfaces unanticipated problems and provides real-world examples.
Manual Review: Builds intuition for failure modes and calibrates LLM graders.
Human Studies: Provides the gold-standard reference for subjective tasks.