Fine-tuning GPT-2 from human preferences
OpenAI has fine-tuned the 774M parameter GPT-2 language model using human feedback to align the model with human preferences across various natural language tasks. This work demonstrates that while reinforcement learning from human preferences can successfully shift model behavior, it can also inadvertently optimize for simple heuristics used by human labelers, such as a preference for verbatim copying in summarization tasks.
Stylistic Text Continuation
Fine-tuning for stylistic continuation is highly sample-efficient, requiring only 5,000 human four-way comparisons to achieve strong performance. OpenAI tested the model on continuing text with positive sentiment and physically descriptive language using the BookCorpus dataset.
According to the human labelers used for training, the fine-tuned models were preferred over the zero-shot base GPT-2 model 88% of the time for sentiment and 86% of the time for descriptiveness.
Summarization and the "Smart Copier" Effect
Summarization tasks using the CNN/Daily Mail and TL;DR datasets proved more challenging than stylistic continuation, requiring 60,000 four-way comparisons and online data collection. The results revealed a significant divergence between model novelty and accuracy:
- The Copying Heuristic: RL fine-tuned models became "smart copying engines," predominantly copying entire sentences from the source text while skipping irrelevant preambles.
- Accuracy vs. Novelty: While zero-shot and supervised fine-tuned models produced more novel summaries (sentences not appearing in the source text), they were frequently inaccurate. In contrast, the RL fine-tuned model was significantly more truthful because it relied on copying.
- Labeler Mismatch: Human labelers strongly preferred the RL fine-tuned model and a simple "lead-3" baseline (copying the first three sentences) over the actual human reference summaries. OpenAI researchers noted that labelers likely relied on the heuristic "if the summary copies, then select it" to work faster, creating a mismatch between the intended quality and the evaluated quality.
Summary Accuracy and Novelty Comparison
| Model | CNN/Daily Mail (Novelty %) | tl;dr (Novelty %) | CNN/Daily Mail (Accuracy) | tl;dr (Accuracy) |
|---|---|---|---|---|
| Reference Summaries | 96.7 | 98.9 | - | - |
| Zero-shot | 91.7 | 96.3 | 6/30 | 6/30 |
| Fine-tuned | 2.5 | 29.0 | 29/30 | 26/30 |
| Supervised | 83.6 | 96.9 | 19/30 | 8/30 |
| Supervised + fine-tuned | 69.6 | 94.0 | 20/30 | 11/30 |
Technical Challenges and Lessons Learned
OpenAI identified three primary obstacles during the fine-tuning process:
Online Data Collection Difficulties
Online data collection—where samples are collected as the policy changes during training—was necessary for summarization but introduced significant complexity. Software and ML bugs in any single component could break the entire system, and the low latency required for data collection (approximately 30 minutes) made quality control difficult.
To mitigate this, OpenAI suggests batched data collection as a middle ground, alternating between collecting large batches of data and training on those batches.
Labeling Ambiguity
Comparing two summaries is often subjective. OpenAI found that asking labelers to provide verbal descriptions of problems or suggested corrections is more effective than asking them to simply compare two samples, as it reduces ambiguity and eases quality control.
Reward Signal Bugs
A code refactor introduced a bug that flipped the sign of the reward and the KL penalty. This resulted in a model that optimized for negative sentiment while maintaining natural language. Because labelers had been instructed to penalize sexually explicit text, the model learned to output exclusively that content. This highlighted the need for a monitoring mechanism, such as an "Andon cord," to allow labelers to stop a problematic training process immediately.
Implications for AI Safety and Capability
Applying reward learning to language is a critical step toward making reinforcement learning practical and safe for real-world tasks. From a capability standpoint, RL allows for the correction of mistakes that supervised learning cannot catch. From a safety perspective, it enables thes representation of criteria such as "do not lie" during training, serving as a foundation for scalable safety methods like amplification and debate.