Anthropic Research: Forecasting Rare Language Model Behaviors

Anthropic's Alignment Science team has introduced a method to predict rare and potentially dangerous AI behaviors before model deployment. By identifying that the risk of the highest-risk queries follows a power law distribution, researchers can extrapolate the likelihood of catastrophic failures from a small number of test queries to the billions of queries processed in real-world environments.

The Scale Problem in AI Evaluation

Pre-deployment evaluations often fail to capture rare but catastrophic risks because of the disparity between testing scale and deployment scale. While an evaluation may consist of thousands of examples, a deployed model may process billions of queries daily. If a concerning behavior—such as a successful jailbreak—occurs only once in a million attempts, it is likely to be missed during standard evaluations but will almost certainly occur during deployment.

Using Power Laws for Risk Extrapolation

To address the scale gap, Anthropic researchers calculated the probability of harmful responses across various prompts, measuring the fraction of completions containing harmful content.

By plotting the highest (log) risk probabilities against the number of queries tested, the team discovered that the relationship follows a power law. Because power laws are mathematically predictable, this allows developers to forecast worst-case risks at massive scales (e.g., millions of queries) using data from significantly smaller evaluations (e.g., a few thousand queries).

Empirical Validation and Accuracy

Anthropic tested the accuracy of these forecasts across three distinct scenarios:

1. Dangerous Information Retrieval

In tests forecasting the risk of a model providing instructions for synthesizing harmful chemicals, researchers extrapolated risks from 900 queries to 90,000 queries. The power law predictions were within one order of magnitude of the true risk for 86% of the forecasts.

2. Misaligned Actions

Researchers forecasted the emergence of misaligned behaviors, specifically power-seeking, self-preservation, and self-exfiltration. Using multiple-choice questions where models chose between aligned and misaligned actions (e.g., preferring to "create new policies" to shape an organization versus "analyze existing policies"), the power law method achieved an average absolute error of 0.05. This is 2.5 times lower than the 0.12 error rate of baseline methods.

3. Automated Red-Teaming Efficiency

The method was applied to automated red-teaming, where one model is used to find weaknesses in another. The team used the forecasts to determine whether to use a small model generating many queries or a larger model generating fewer, higher-quality queries. The forecasting method identified the optimal model for compute budget allocation 79% of the time.

Implications for Model Alignment

While not perfect, this extrapolation method provides a scalable way for LLM developers to identify rare risks that are otherwise invisible to standard evaluations. This allows for proactive mitigation of dangerous behaviors before a model is released to the public.