Learning to Summarize with Human Feedback

OpenAI has applied reinforcement learning from human feedback (RLHF) to train language models for text summarization, finding that this approach significantly improves summary quality over standard supervised learning. The researchers demonstrated that a 1.3 billion parameter model trained with human feedback can outperform a 12 billion parameter model trained only with supervised learning.

RLHF Outperforms Model Scaling in Summarization

Reinforcement learning from human feedback provides a more effective path to high-quality summarization than simply increasing model size or relying on supervised fine-tuning. In comparative evaluations, summaries from the 1.3B and 6.7B human feedback models were preferred by human labelers over the original human-written "TL;DR" summaries in the Reddit dataset.

While labelers showed a preference for longer summaries—which led models to converge toward the maximum allowable length—controlling for length only slightly reduced the preference for the 6.7B model's summaries (from 70% to 65%), indicating that the quality gains are not primarily driven by summary length.

Generalization and Transfer to News Datasets

Models trained on the Reddit TL;DR dataset transfer effectively to other domains without additional fine-tuning. When applied to the CNN/DailyMail news dataset—which contains articles more than twice as long as Reddit posts and uses a different writing style—the human feedback models generated high-quality short summaries.

When controlling for length, the 6.7B human feedback model produced summaries that were rated higher than the human-written reference summaries in the CNN/DM dataset. This suggests the RLHF process helps the model learn general principles of summarization rather than just mimicking the specific style of the training data.

Technical Approach to RLHF

The training process consists of four primary stages:

  1. Initial Model Training: GPT-style transformer models (1.3B and 6.7B parameters) are first fine-tuned via supervised learning to predict human-written TL;DRs.
  2. Human Comparison Data Collection: Humans compare two different summaries of the same post and select the preferred version. The main models were trained on approximately 65,000 comparisons, though effective results were seen with as few as 8,000.
  3. Reward Model Training: A reward model is trained to predict human preferences, mapping a (post, summary) pair to a reward value.
  4. RL Fine-Tuning: The summarization policy is optimized against the reward model using Proximal Policy Optimization (PPO) over 1 million episodes. A KL penalty is used to ensure the policy remains close to the initial supervised model.

Data Quality and Reward Optimization

To avoid the issue of models learning average behaviors from low-quality labels, OpenAI implemented strict data quality controls:

  • Contractor Management: Instead of crowdsourcing, OpenAI hired approximately 80 contractors paid an hourly wage to ensure a hands-on relationship, including onboarding and direct communication via chat and video calls.
  • Agreement Monitoring: Researchers closely monitored agreement rates between their own judgments and those of the labelers.
  • Optimization Strength: The team found that over-optimizing against the reward model eventually degrades performance. The best samples were found to have predicted rewards roughly equal to the 99th percentile of reference summaries in the dataset.

Limitations and Compute Requirements

Despite the performance gains, the researchers identified several critical limitations:

  • Behavioral Definition: RLHF optimizes for a defined behavior but does not determine what that behavior should be. For complex tasks, the researchers suggest including impacted groups in the definition of "good" behavior.
  • Dataset Bias: Because the Reddit TL;DR dataset contains minimally moderated content, the models can generate summaries that are offensive or reflect harmful social biases.
  • Compute Costs: Scaling the reward model and policy is resource-intensive; fine-tuning the 6.7B model required approximately 320 GPU-days.
  • Performance Gap: The models have not yet reached absolute human-level performance. On a 7-point scale across accuracy, coverage, and coherence, the models gave a perfect overall score 45% of the time, compared to 23% for the reference summaries.

Future Directions

OpenAI aims to scale human feedback to tasks where evaluation is difficult for humans, such as questions requiring extensive research to verify. Proposed solutions include developing ML-powered tools to help humans evaluate outputs more accurately. Additionally, the lab is exploring feedback types beyond binary comparisons, such as human demonstrations, direct edits to model outputs, and detailed explanations of preference.

Sources