Deliberative alignment: reasoning enables safer language models

Overview

Deliberative alignment teaches reasoning models the text of safety specifications and trains them to reason over these specifications before answering, producing safer and appropriately calibrated responses.

Method

The approach first trains an o‑style model for helpfulness without safety data, then builds a supervised fine‑tuning dataset where chain‑of‑thought completions reference the relevant safety specification text inserted into the system prompt, filters the data with a policy‑aware reward model, and finally uses reinforcement learning with that same reward model to improve the model’s use of chain‑of‑thought for safety reasoning.

Results

o1 models achieve a Pareto improvement on both under‑ and over‑refusals compared to GPT‑4o and other state‑of‑the‑art LLMs, saturating many internal and external safety benchmarks such as StrongREJECT and XSTest, and show strong generalization to out‑of‑distribution safety scenarios.

Conclusion

Deliberative alignment demonstrates how advances in model capabilities can be harnessed to improve AI safety, providing finer‑grained control over compliance, refusal, and safe completion while reducing reliance on human‑labeled data.