Constitutional AI: Harmlessness from AI Feedback

Anthropic has developed Constitutional AI, a framework for training AI assistants to be harmless without relying on extensive human labeling of harmful outputs. By using a set of predefined rules or principles—a "constitution"—the system improves its own behavior through a two-phase process of supervised learning and reinforcement learning from AI feedback (RLAIF).

The Constitutional AI Process

Constitutional AI replaces human-led reward modeling with AI-driven evaluation based on a set of principles. The training process is divided into two primary phases:

Supervised Learning (SL) Phase

In the supervised phase, the model generates initial responses to prompts. It then generates self-critiques and revisions of those responses based on the following workflow:

Sampling: The initial model samples a response.
Critique: The model generates a critique of its own response based on the constitution.
Revision: The model revises the response to be more aligned with the principles.
Finetuning: The original model is finetuned on these AI-generated revised responses.

Reinforcement Learning (RL) Phase

Following the supervised phase, the model undergoes Reinforcement Learning from AI Feedback (RLAIF). In this RL phase, the system uses a model to evaluate which of two sampled responses is better according to the constitution. This creates a dataset of AI preferences, which is used to train a preference model. This preference model then serves as the reward signal for the final RL training step.

Key Capabilities and Outcomes

Constitutional AI enables the creation of an AI assistant that is both harmless and non-evasive. Unlike models that may simply refuse to answer a harmful query, a Constitutional AI assistant engages with harmful queries by explaining its objections to them, providing transparency into its decision-making process.

To improve performance and transparency, both the Supervised Learning and RL phases leverage chain-of-thought style reasoning. This allows the model to articulate the reasoning behind its critiques and revisions, making the AI's decision-making process more interpretable to humans.

Implications for AI Safety and Control

Constitutional AI demonstrates that AI behavior can be controlled more precisely with significantly fewer human labels. By shifting the oversight from labeling individual outputs to defining the high-level rules (the constitution), researchers can scale the safety training of AI systems as they become more capable, using the AI's own capabilities to supervise other AIs.