Anthropic Research: The Capacity for Moral Self-Correction in LLMs

Anthropic has demonstrated that Large Language Models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) possess the capacity for "moral self-correction." This ability allows models to avoid producing harmful outputs when explicitly instructed to do so, providing a pathway for training models to abide by ethical principles.

Emergence of Moral Self-Correction at 22B Parameters

Language models exhibit the capability for moral self-correction starting at 22B model parameters. This capability typically improves as model size increases and as RLHF training is applied.

Anthropic identifies two primary drivers for this emergence:

1. Instruction Following: The model's ability to accurately follow complex prompts.
2. Normative Concept Learning: The ability to learn complex normative concepts of harm, such as bias, discrimination, and stereotyping.

By combining these two capabilities, models at this scale can follow instructions to avoid specific types of morally harmful outputs.

Technical Framework for Moral Self-Correction

Moral self-correction is defined as the ability of a model to avoid producing harmful outputs if instructed to do so. The research conducted across three different experiments to reveal different facets of this capability.

The findings suggest that the capacity for self-correction is not an inherent trait of all LLMs, but is a result of thescalars

Implications for AI Safety and Ethics

The results of this research indicate a cautious optimism regarding the ability to train language models to adhere to ethical principles. Because models can learn to identify and avoid harm based on instructions, the ability to actually implement safety guardrails through instruction-tuning and RLHF is technically feasible at scale.