Anthropic: Training a Helpful and Harmless Assistant with RLHF

Anthropic has developed a method to fine-tune language models into helpful and harmless assistants using preference modeling and Reinforcement Learning from Human Feedback (RLHF). This alignment process enhances performance across nearly all NLP evaluations and remains compatible with training for specialized tasks like summarization and Python coding.

RLHF for Model Alignment and Performance

Reinforcement Learning from Human Feedback (RLHF) is used to align language models with human preferences, specifically focusing on the dual goals of being helpful and harmless. This alignment training does not trade off general capability for safety; rather, it improves performance on almost all NLP evaluations. Furthermore, RLHF is fully compatible with the acquisition of specialized skills, such as Python coding and summarization, indicating that alignment can coexist with high-level technical proficiency.

Iterated Online Training Methodology

To improve datasets and models efficiently, Anthropic employs an iterated online mode of training. In this approach, preference models and RL policies are updated on a weekly cadence using fresh human feedback data. This iterative cycle allows the model to evolve based on current performance and new human inputs, creating a continuous loop of refinement.

Robustness and RL Reward Analysis

Research into the robustness of RLHF training reveals a specific mathematical relationship between the reward and the policy's divergence from its starting point. Specifically, there is a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization.

Additional Technical Analyses

Beyond the primary RLHF results, the research includes several peripheral analyses to ensure model stability and reliability:

• Calibration and Objectives: Investigation into how the model's confidence aligns with accuracy and how competing objectives (such as helpfulness vs. harmlessness) are balanced.
• OOD Detection: The use of Out-of-Distribution (OOD) detection to identify prompts the model is not equipped to handle.
• Human Comparison: Comparison of model outputs against those produced by human writers to benchmark quality and tone.
• Prompt Testing: Evaluation of the models using prompts derived from recent related research to test consistency and safety.