OpenAI InstructGPT: Aligning Language Models to Follow Instructions

OpenAI has developed InstructGPT, a set of language models designed to align more closely with user intentions than GPT-3. By utilizing reinforcement learning from human feedback (RLHF), InstructGPT improves instruction-following capabilities, increases truthfulness, and reduces the generation of toxic content.

Improved Instruction Following and Safety

InstructGPT significantly outperforms GPT-3 in following English instructions and providing helpful outputs. While GPT-3 was trained to predict the next word on a large dataset of internet text—often leading to outputs that were untruthful or toxic—InstructGPT is specifically aligned to perform the tasks users actually want.

Key performance improvements include:

  • Higher User Preference: Human labelers significantly prefer InstructGPT outputs over GPT-3, even when GPT-3 is provided with few-shot prompts to enter an "instruction-following mode."
  • Efficiency in Scale: Labelers preferred the 1.3B parameter InstructGPT model over the 175B parameter GPT-3 model, despite the former having more than 100x fewer parameters.
  • Reduced Toxicity and Falsehoods: According to the RealToxicityPrompts and TruthfulQA benchmarks, InstructGPT produces fewer imitative falsehoods and is less toxic than GPT-3.
  • Lower Hallucination Rates: Human evaluations on API prompt distributions show that InstructGPT makes up facts less frequently and generates more appropriate responses.

The RLHF Training Methodology

InstructGPT is trained using a three-step reinforcement learning from human feedback (RLHF) process to "unlock" capabilities already present in the pretrained GPT-3 model:

  1. Supervised Fine-Tuning: Labelers provide demonstrations of desired model behavior on prompts submitted to the API.
  2. Reward Model Training: Labelers rank several model outputs, and a reward model (RM) is trained to predict which output a human would prefer.
  3. PPO Fine-Tuning: The GPT-3 policy is fine-tuned using the PPO algorithm to maximize the reward signal provided by the RM.

To prevent the "alignment tax"—where performance on academic NLP tasks drops as the model aligns with customer preferences—OpenAI mixed a small fraction of the original GPT-3 pretraining data into the RL fine-tuning process using normal log likelihood maximization. This approach maintains safety and preference alignment while preserving or even improving performance on academic benchmarks.

Generalization and Preference Alignment

OpenAI tested whether InstructGPT overfit to the specific preferences of its training labelers. Experiments showed that held-out labelers (those who did not produce training data) preferred InstructGPT outputs at the same rate as the training labelers. Additionally, reward models trained on a subset of labelers generalized well to other subsets, suggesting the model aligns with broader preferences rather than just a specific group of annotators.

Current Limitations and Safety Risks

Despite these advancements, InstructGPT is not fully aligned or safe. OpenAI identifies several ongoing challenges:

  • Residual Risks: The models still generate biased, toxic, or sexual and violent content without explicit prompting.
  • Susceptibility to Misuse: Because the models are better at following instructions, they may be more susceptible to producing unsafe outputs if explicitly instructed to do so. Reliable refusal of harmful instructions remains an open research problem.
  • Cultural Bias: Since the model is trained on English instructions, it is biased toward the cultural values of English-speaking populations.

To mitigate these risks, OpenAI continues to review potential applications, provide content filters for detecting unsafe completions, and monitor for misuse.

Sources