OpenAI Announces Alignment Strategy Using RL from Human Feedback and InstructGPT

TL;DR

OpenAI announced that Reinforcement Learning from Human Feedback (RLHF) and the InstructGPT family are its primary tools for aligning deployed language models, achieving strong human preference with less than 2 % of GPT‑3’s pre‑training compute, but the models still have notable shortcomings and the lab is planning to use increasingly capable models to automate alignment research.

RL from Human Feedback as the Core Alignment Technique

OpenAI states that RL from human feedback (RLHF) is the main technique used to align its deployed language models today. By fine‑tuning pretrained models such as GPT‑3 with human‑generated preference data, OpenAI creates a class of models called InstructGPT that are trained to follow both explicit instructions and implicit intents like truthfulness, fairness, and safety.

Efficiency and Performance of InstructGPT

  • Human Preference: InstructGPT is preferred by users over a pretrained model that is 100× larger.
  • Compute Cost: Fine‑tuning InstructGPT consumes < 2 % of the compute required for GPT‑3’s original pre‑training.
  • Human Feedback Effort: The fine‑tuning process involved roughly 20,000 hours of human feedback.
  • Customer Preference: In practice, OpenAI’s API customers already favor InstructGPT over the raw pretrained models.

These results demonstrate that substantial alignment gains can be achieved with relatively modest additional resources.

Current Limitations of InstructGPT

OpenAI acknowledges that present versions of InstructGPT are far from fully aligned. Specific failure modes include:

  • Inability to follow simple instructions reliably.
  • Inconsistent truthfulness.
  • Failure to refuse harmful requests.
  • Occasional biased or toxic outputs.
  • Reduced creativity compared to the base pretrained models, a shortcoming not captured by public benchmarks.

The lab is actively researching a deeper scientific understanding of RLHF and ways to improve the quality of human feedback.

Alignment Research Infrastructure

OpenAI’s natural‑language API serves as a feedback loop for alignment experiments, exposing the models to a diverse set of real‑world tasks that customers actually pay for. This environment enables rapid assessment of alignment techniques in practice.

Scaling Alignment Beyond Current Models

  • Distinction from AGI Alignment: Aligning the API is easier than aligning artificial general intelligence (AGI) because the tasks are human‑supervisable and the models are not smarter than humans.
  • Role of RLHF: While RLHF alone is unlikely to solve AGI alignment, it is considered a core building block for scalable alignment proposals.
  • No Known Indefinitely Scalable Solution: OpenAI notes that a forever‑scalable alignment method has not been discovered, and new alignment challenges are expected as AI capabilities grow.

Pragmatic Roadmap: AI‑Assisted Alignment Research

OpenAI proposes a pragmatic approach:

  1. Build systems that accelerate alignment research faster than humans can.
  2. Allow AI systems to take over more alignment tasks, eventually conceiving, implementing, and evaluating new alignment techniques.
  3. Shift human effort toward reviewing AI‑generated alignment work rather than producing it.
  4. Target narrower AI systems with human‑level competence in relevant domains, which are expected to be easier to align than general‑purpose or superhuman systems.

Why Language Models Are Suited for Alignment Automation

  • They contain extensive knowledge of human values from internet text.
  • They are not independent agents and do not pursue autonomous goals.
  • Alignment tasks can often be expressed as natural‑language or coding problems, fitting the models’ strengths.

Future Alignment Assistants

OpenAI mentions that upcoming versions of WebGPT, InstructGPT, and Codex could serve as alignment research assistants, though they are not yet capable enough for meaningful contributions. The lab plans to train a model that can be useful for alignment research and make it available to the external alignment community once it reaches sufficient capability.

This post faithfully reflects OpenAI’s August 2022 announcement on its alignment research strategy, without adding or extrapolating beyond the original text.

Sources