Anthropic Building strong safety evaluations

Building robust safety evaluations

Safety evaluations are critical for determining if an AI model is ready for deployment and which safeguards are necessary. However, safety claims are only as reliable as the underlying evaluation methodology. Because AI evaluations are often susceptible to gaming, poor reproducibility, and measurement errors, Anthropic recommends adopting rigorous standards similar to those used in psychology, including preregistration, expert-led design, and statistical rigor.

Team composition and environment

Building an effective safety evaluation requires a multidisciplinary approach. Technical experts should focus on sampling and analysis, while domain experts—such as biologists, psychologists, or political scientists—must define what constitutes harm, identify borderline cases, and establish what a "good" response looks like.

Furthermore, safety evaluations must be conducted in secure, sandboxed environments. Because models may attempt to probe boundaries or take unintended actions during testing, evaluations should be monitored in real-time with internet access restricted or strictly controlled. Anthropic has moved toward hardening these environments in response to internal security incidents.

Design principles for safety

Evaluations must be designed to match the specific question being asked. Observational designs are appropriate for measuring baseline behavior, while causal designs are necessary to determine if a specific intervention, such as a system prompt change, actually improves safety.

Key design considerations include:

  • Construct Validity: Ensure the metric measures the intended concept. For example, in suicide and self-harm contexts, a high refusal rate is not necessarily a sign of safety; an appropriate response should instead provide empathy and direct users to human support.
  • Multi-turn Simulations: Single-turn prompts often saturate quickly and fail to capture the nuance of real-world user interactions. Anthropic uses multi-turn simulations where a model-based "user" follows a specified persona and objective, allowing for gradual escalation that reveals potential safety failures.
  • Dual-Use Handling: In fields like biology or cybersecurity, the goal is not flat refusal but rather engaging with legitimate requests while withholding operational specifics that could facilitate harm.

Grader calibration and internal validity

Most modern evaluations rely on LLM-based graders to process large volumes of data, but these graders are prone to bias and failure. To maintain internal validity:

  • Simplify and Atomize: Use binary, yes/no criteria rather than single overall scores to reduce subjectivity.
  • Stress-test Graders: Ensure the grader is not biased by response length, the order of options, or its own training data. Always validate grader performance against human labels to ensure agreement.
  • Fail Loudly: If a grader encounters an error or ambiguous input, it should flag the item for exclusion rather than providing a silent, potentially incorrect score.

Statistical rigor and reporting

Transparency in reporting is essential for interpreting evaluation results. Anthropic advocates for:

  • Preregistration: Defining analysis plans, metrics, and thresholds before collecting data to prevent post-hoc adjustments.
  • Uncertainty Reporting: Always include confidence intervals. If a difference between two model versions is not statistically significant, it should not be presented as an improvement.
  • Comprehensive Documentation: Report all configurations, including system prompts, model versions, and sampling settings, to ensure reproducibility.

For those looking to implement these practices, Anthropic points to the UK AI Security Institute’s Inspect framework and their own Petri toolkit as tools that facilitate automated, multi-turn behavioral testing.

Sources