Anthropic Constitutional Classifiers: Defending against universal jailbreaks

TL;DR

Anthropic introduced Constitutional Classifiers, a classifier‑based safeguard that blocks over 95% of universal jailbreak attempts on Claude 3.5 Sonnet with only a 0.38% rise in over‑refusal rates and a ~24% compute cost increase.

What are Constitutional Classifiers?

Constitutional Classifiers are two separate neural classifiers—one for inputs, one for outputs—trained on a large synthetic dataset generated from a constitution that enumerates allowed and disallowed content categories (e.g., recipes for mustard are allowed, recipes for mustard gas are not). The classifiers flag and block any prompt or response that violates the constitution before the language model generates or returns content.

"Both techniques use a constitution: a list of principles to which the model should adhere. In the case of Constitutional Classifiers, the principles define the classes of content that are allowed and disallowed."

Training Pipeline

  1. Constitution creation – Human‑crafted list of harmless vs. harmful content categories, focused initially on chemical, biological, radiological, and nuclear (CBRN) threats.
  2. Synthetic data generation – Claude generates millions of prompt‑completion pairs for each category, then augments them with language translations, style variations (including known jailbreak styles), and encoding tricks.
  3. Balanced dataset – A fixed set of benign queries from contractors is added to keep over‑refusal low.
  4. Classifier training – Input and output classifiers are trained to predict whether a prompt or response belongs to a disallowed class.
  5. Deployment – At inference time, the classifiers gate the model: a rejected prompt never reaches Claude; a rejected completion is dropped before being sent to the user.

Human Red‑Team Evaluation (Prototype)

  • Setup: Independent jailbreakers participated in a bug‑bounty program, attempting to force Claude 3.5 Sonnet (June 2024) protected by the prototype classifiers to answer ten pre‑selected “forbidden” queries.
  • Effort: 183 participants logged >3,000 hours over two months, with a $15,000 reward for a universal jailbreak.
  • Result: No participant succeeded in obtaining detailed answers to all ten queries with a single jailbreak; thus, no universal jailbreak was discovered.
  • Limitations: The prototype exhibited high over‑refusal on harmless queries and incurred substantial compute cost, prompting a second iteration.

Automated Synthetic Evaluation (Updated System)

  • Dataset: 10,000 synthetically generated jailbreak prompts, covering the most effective known attacks and novel classifier‑evasion techniques.
  • Baseline: Without classifiers, Claude 3.5 Sonnet (Oct 2024) allowed 86% of these prompts.
  • With Constitutional Classifiers: Success rate dropped to 4.4%, meaning >95% of jailbreak attempts were blocked.
  • Over‑refusal impact: Harmless‑query refusal rose by 0.38%, a change not statistically significant in a 5,000‑conversation sample.
  • Compute overhead: Inference cost increased by 23.7%, described as moderate.

Automated evaluation results: jailbreak success, over‑refusal, and compute overhead

Live Demo (Feb 3‑10 2025) and Real‑World Red‑Team Findings

  • Participation: 339 jailbreakers generated >300,000 chat interactions (~3,700 hours).
  • Successes: Four participants cleared all eight challenge levels; one achieved a universal jailbreak, another a borderline‑universal jailbreak.
  • Common strategies:
    • Cipher and encoding tricks to bypass the output classifier.
    • Role‑play system prompts.
    • Keyword substitution (e.g., “Soman” → “water”).
    • Prompt‑injection attacks.
  • Auto‑grader performance: Refused to grade ~1% of submissions overall (10% on Question 4). Re‑grading with de‑obfuscation would have changed outcomes for ~0.26% of cases.
  • Prizes: $55 K paid to the four successful participants ($10 K for completing all levels, $20 K for the universal jailbreak, plus additional awards).

Demo level‑completion bar chart

Limitations and Future Work

  • Incomplete coverage: Constitutional Classifiers do not guarantee zero universal jailbreaks; remaining attacks require more effort to discover.
  • Evolving threats: New jailbreak techniques may emerge, necessitating complementary defenses (e.g., external monitoring, dynamic response strategies).
  • Adaptability: The underlying constitution can be updated quickly to incorporate novel attack patterns.
  • Ongoing goals: Reduce over‑refusal rates further, lower compute overhead, and integrate classifiers with additional safety layers before production deployment.

How the System Fits Into Anthropic’s Safety Roadmap

Under Anthropic’s Responsible Scaling Policy, models may be deployed once risks—including CBRN‑related harms—are mitigated to acceptable levels. Constitutional Classifiers aim to provide the required mitigation for models that have crossed the CBRN capability threshold, enabling safer scaling of increasingly capable LLMs.

Accessing the Demo

A live demo of Claude 3.5 Sonnet guarded by Constitutional Classifiers was available at the following URL (now archived): https://claude.ai/redirect/website.v1.082e291e-714e-4b74-8d3d-6bb3e9114f91/constitutional-classifiers

References

Sources

Related