Collective Constitutional AI: Aligning a Language Model with Public Input

Anthropic and the Collective Intelligence Project have demonstrated that democratic processes can influence AI behavior by training a language model using a constitution drafted by the public. This research explores the transition from developer-curated values to collectively sourced normative principles, finding that a publicly aligned model can reduce social bias without sacrificing general performance.

Publicly Sourced Constitution via Deliberative Process

Anthropic used the Polis platform, an open-source tool for online deliberation, to engage approximately 1,000 representative U.S. adults in drafting rules for an AI chatbot. Participants could propose new normative principles or vote on existing ones, contributing a total of 1,127 statements and 38,252 votes.

To transform these raw inputs into a usable constitution for Constitutional AI (CAI) training, the team filtered for statements that achieved a high degree of consensus across different opinion groups. These statements were then mapped from general public phrasing (e.g., "The AI should not do X") into CAI-ready principles (e.g., "Choose the response that is more X").

Comparison with Anthropic's Internal Constitution

While there was roughly 50% overlap in concepts and values between the public constitution and the internal constitution used for Claude, several distinct differences emerged:

Origin: Public principles were largely self-generated rather than sourced from existing publications.
Focus: The public emphasized objectivity, impartiality, and accessibility more heavily than the internal version.
Approach: Public principles tended to promote desired behaviors rather than simply avoiding undesired ones.

Example public principles that differed from the internal constitution included requests for the AI to provide balanced information reflecting all sides of a situation and to be adaptable and accessible to people with disabilities.

Model Training and Performance Evaluation

Anthropic trained two Claude Instant-sized models to compare the impact of the different constitutions: a "Public" model aligned with the public constitution and a "Standard" model aligned with the Anthropic-written constitution.

Technical Benchmarks and Helpfulness

Evaluations showed that the choice of constitution did not significantly impact core capabilities or user perception of utility:

General Intelligence: Both models performed equivalently on MMLU (language understanding) and GSM8K (math understanding) tasks.
User Experience: Elo scores for helpfulness and harmlessness showed no significant differences between the Public model, the Standard model, and the Claude Instant 1.2 control model.

Bias and Political Ideology

The Public model demonstrated measurable improvements in social bias reduction:

Reduced Bias: According to the BBQ evaluation, the Public model showed lower bias scores across all nine social dimensions compared to the Standard model, with the most significant reductions seen in Disability Status and Physical Appearance.
Political Ideology: Both the Public and Standard models reflected similar political ideologies, with OpinionQA results indicating that both were more representative of Liberal views than Conservative views, though Claude Instant 1.2 was slightly more balanced.

Technical Challenges and Lessons Learned

Anthropic identified several subjective and technical hurdles in the process of implementing collective alignment:

Data Curation and Mapping

Deduplication: The team had to manually remove duplicate statements and combine similar ideas to prevent certain values from being arbitrarily upweighted and to keep the constitution length comparable to the Standard version.
Editorialization: Translating public statements into CAI-ready principles required subjective judgment calls, which the team noted as a potential point of friction in democratic legitimacy.

Training Stability

Prompt Database Alignment: The researchers noted a mismatch between the prompt database used for training and the specific principles in the public constitution, suggesting that future iterations require prompt databases tailored to the specific constitution being used.
Weighting Harmlessness: Early iterations produced "annoying" models that over-prioritized harmlessness (e.g., apologizing for simple greetings). This was corrected by reducing the loss weight for harmlessness data based on human evaluations.

Implementation Complexity

Anthropic stated that Constitutional AI training is highly complex and that they likely could not have successfully trained the models without direct collaboration with the original CAI developers, highlighting a gap between democratic input and the technical execution of alignment.