Anthropic A Statistical Approach to Model Evaluations

Anthropic has introduced a new research paper, "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations," which provides a framework for reporting AI model evaluation results with scientific rigor. The core objective is to determine whether a model's superior performance on a benchmark is a result of genuine capability or simply the "luck of the draw" regarding the specific questions selected.

Quantifying Skill via the Central Limit Theorem

To measure underlying skill independent of specific question sets, researchers should focus on the theoretical average across all possible questions rather than the observed average of a specific benchmark.

Under the Central Limit Theorem, mean values of random samples from the same distribution follow a normal distribution. Anthropic recommends reporting the Standard Error of the Mean (SEM) alongside every eval score. This allows for the calculation of a 95% confidence interval by adding and subtracting 1.96 SEM from the mean score, providing a mathematically grounded measure of a model's theoretical performance.

Addressing Non-Independent Questions with Clustered Standard Errors

Many popular benchmarks, such as SQuAD, RACE, QuAC, and DROP, violate the assumption of independent questions because they group multiple questions around a single passage of text. Applying a naive Central Limit Theorem approach to these clustered questions underestimates the standard error, which can lead researchers to falsely detect capability differences where none exist.

Anthropic recommends clustering standard errors on the unit of randomization (e.g., the text passage). In practical tests, Anthropic found that clustered standard errors on popular evals can be over three times larger than naive standard errors.

Strategies for Reducing Variance Within Questions

Reducing the variance of the random component of a score directly increases the statistical precision of the overall mean. Anthropic suggests two primary strategies based on the prompting method:

For Chain-of-Thought (CoT) Reasoning: Researchers should resample answers from the same model multiple times and use question-level averages as the input for the Central Limit Theorem. The Inspect framework currently implements this via its epochs parameter.
For Non-CoT Reasoning: The random component can often be eliminated by using next-token probabilities. For example, in a multiple-choice question where "B" is the correct answer, the score should be the probability the model assigns to the token "B" rather than a binary correct/incorrect result.

Improving Signal through Paired-Difference Analysis

Comparing two models using a two-sample t-test ignores the shared structure of the data. Because models are tested on the same list of questions, a paired-differences test is more effective as it eliminates variance caused by question difficulty and focuses solely on the variance in responses.

Anthropic notes that frontier models show a substantial correlation in their question scores (between 0.3 and 0.7), meaning they tend to get the same questions right or wrong. Reporting mean differences, standard errors, confidence intervals, and correlations provides a clearer signal of relative performance.

Using Power Analysis for Eval Design

Statistical power is the ability of a test to detect a difference between two models if one actually exists. Without sufficient power, small but real differences in capabilities may go undetected.

Anthropic recommends using power analysis to:

Formulate specific hypotheses (e.g., "Model A outperforms Model B by 3 percentage points").
Calculate the required number of questions to test that hypothesis against a null hypothesis (e.g., "Model A and Model B are tied").
Determine the necessary number of resamples for answers to maintain desired power properties.
Decide if a limited-question eval is worth running based on the expected effect size.