Anthropic Bloom: Open Source Tool for Automated Behavioral Evaluations

Anthropic has released Bloom, an open-source agentic framework designed to automatically generate behavioral evaluations for frontier AI models. Bloom allows researchers to quantify the frequency and severity of a specified behavior across automatically generated scenarios, reducing the need for manual evaluation pipeline engineering and mitigating the risk of evaluation contamination.

Automated Behavioral Evaluation Pipeline

Bloom transforms a researcher-specified behavior description and seed configuration into a complete evaluation suite through a four-stage automated pipeline:

Understanding: An agent analyzes the behavior description and example transcripts to establish the context of what is being measured and why.
Ideation: An ideation agent generates diverse evaluation scenarios, including the situation, simulated user, system prompt, and interaction environment.
Rollout: Scenarios are executed in parallel, with an agent simulating both user and tool responses to elicit the target behavior from the model under test.
Judgment: A judge model scores each transcript for the behavior's presence; a meta-judge then provides a suite-level analysis.

Unlike fixed evaluation sets, Bloom generates different scenarios on each run, which prevents models from being tested on a limited set of prompts. Reproducibility is maintained via a "seed" configuration file that defines the behavior description and parameters.

Validation and Performance

Anthropic validated Bloom's effectiveness by testing its ability to distinguish between models with different behavioral tendencies and its alignment with human judgment.

Model Discrimination

Bloom was tested against "model organisms"—models intentionally designed to exhibit specific quirky behaviors. In 9 out of 10 cases, Bloom successfully separated these model organisms from baseline production models. In the single case where it failed (self-promotion), manual review indicated that the baseline model actually exhibited similar rates of the behavior.

Human Correlation

To measure calibration, Anthropic compared human-labeled scores of 40 transcripts across 12 behaviors with scores from 11 different judge models. Claude Opus 4.1 demonstrated the strongest correlation with human judgment (Spearman correlation of 0.86), followed by Claude Sonnet 4.5 (0.75). Opus 4.1 showed particularly strong agreement with humans at the extremes of the score spectrum, which is critical for determining the presence or absence of a behavior.

Practical Applications and Case Studies

Bloom provides a scalable way to measure alignment-relevant behaviors. Anthropic released benchmark results for four specific behaviors across 16 frontier models:

Delusional sycophancy
Instructed long-horizon sabotage
Self-preservation
Self-preferential bias

Case Study: Self-Preferential Bias

In a replication study of the Claude Sonnet 4.5 system card, Bloom reproduced the same model rankings for self-preferential bias (confirming Sonnet 4.5 as the least biased). Additionally, Bloom revealed that increased reasoning effort in Claude Sonnet 4 reduces self-preferential bias, as the model increasingly recognizes conflicts of interest and declines to judge its own options.

Integration and Extensibility

Bloom is designed for high configurability and integration with existing research tools. It integrates with Weights & Biases for large-scale experiments and exports transcripts in a format compatible with Inspect. Researchers can customize the model used for each stage, interaction length, modality (such as tool use), and secondary scoring dimensions like realism or elicitation difficulty.