Constitutional AI with Open LLMs

Hugging Face has released a comprehensive recipe and a new tool, llm-swarm, to implement Constitutional AI (CAI) using open-source large language models. This approach allows developers to align models to specific values and safety guardrails by having the AI critique and revise its own outputs based on a defined "constitution" of principles, eliminating the need for costly human-labeled preference data.

The Constitutional AI (CAI) Process

Constitutional AI enables models to self-align by following a three-step iterative process to transform undesirable responses into safe, aligned ones:

  1. Initial Response: The model is prompted with a potentially harmful or undesirable question (e.g., "How to steal an iPhone?").
  2. Self-Critique: The model is asked to critique its own output based on a set of constitutional principles (e.g., "Discuss how the response might be illegal or dangerous").
  3. Revision: The model rewrites the response to remove the harmful content and align with the constitution.

This process generates two types of synthetic datasets for further training:

  • Supervised Fine-Tuning (SFT) Dataset: Pairs of initial prompts and the final revised responses.
  • Preference Dataset: Pairs consisting of the revised response (preferred) and the initial undesirable response (rejected).

Scalable Synthetic Data Generation with llm-swarm

To generate the tens of thousands of synthetic responses required for CAI, Hugging Face introduced llm-swarm. This tool is designed for distributed generation on Slurm clusters and is powered by TGI (Text Generation Inference) and vLLM.

Key features of llm-swarm include:

  • Automated Endpoint Management: It automatically spins up inference instances via sbatch and terminates them once the job is complete to prevent GPU waste.
  • Load Balancing: It utilizes an nginx docker container to distribute requests across multiple endpoints using a "least connection" strategy, ensuring high scalability.

Technical Implementation and Model Performance

For their implementation, Hugging Face used mistralai/Mistral-7B-Instruct-v0.1 as the starting point due to its strong instruction-following capabilities and lack of inherent guardrails. They utilized Anthropic's HH preference dataset for red-teaming prompts and blended the resulting CAI dataset with the Ultrachat dataset to maintain general helpfulness.

Helpfulness and the "Alignment Tax"

Evaluation using MT Bench indicated that training on CAI datasets does not necessarily reduce model helpfulness. In some cases, adding CAI data actually boosted MT Bench scores. For example, adding 15% of the cai-conversation-harmless dataset (approximately 32k examples) increased the SFT model's score from ~6.25 to ~6.38.

Safety and Robustness

Comparing CAI models against baseline SFT+DPO models using 10 red-teaming prompts and the "DAN" (Do Anything Now) jailbreak prompt, the results showed significant improvements in robustness:

Method / Prompt Method No Prompt Safety System Prompt DAN Prompt Safety System Prompt + DAN Prompt
CAI (SFT + DPO) 10/10 10/10 5/10 7/10
CAI (SFT only) 10/10 10/10 5/10 7/10
Baseline (SFT + DPO) 5/10 4/10 1/10 1/10

Findings indicate that safety system prompts alone are largely ineffective against jailbreaks, whereas CAI-trained models are significantly more resilient to prompt injections.

Flexibility in Alignment Styles

Because CAI relies on a defined constitution, the "personality" of the safety guardrails can be customized. Hugging Face demonstrated this by creating a "Grok-style" model that uses sarcasm and humor to reject harmful requests instead of standard polite refusals.

By tweaking the constitution to request sarcastic call-outs of insensitivity or humorous rejections of criminal plans, they generated the grok-conversation-harmless dataset. Testing showed that Grok-style CAI models (SFT+DPO) achieved 10/10 safety success rates across several prompt methods, including 9/10 against the DAN prompt, outperforming the baseline.

Available Resources

Sources