OpenAI Summarizing Books with Human Feedback (2021)

Approach: Reinforcement Learning from Human Feedback and Recursive Task Decomposition

OpenAI's method uses reinforcement learning from human feedback to train a model that summarizes books by first summarizing small sections and then recursively summarizing those summaries into higher-level summaries. As the post states, "Our model works by first summarizing small sections of a book, then summarizing those summaries into a higher-level summary, and so on."

Motivation: Scaling Human Oversight for the Alignment Problem

The work tackles the alignment problem by developing scalable oversight techniques for tasks where human evaluation is hard, such as book summarization. The post explains, "To safely deploy powerful, general-purpose artificial intelligence in the future, we need to ensure that machine learning models act in accordance with human intentions. This challenge has become known as the alignment problem."

Methodology: Combining RLHF with Recursive Decomposition

The approach integrates reinforcement learning from human feedback to align model outputs with human preferences, applying it hierarchically across sections of a book. The post notes that they trained a model to summarize entire books using this technique.

Example: Demonstrations on Classic Books

The post shows sample summaries for books including Alice in Wonderland, Pride and Prejudice, Romeo and Juliet, Twelve Years a Slave, and The Story of My Life, with the original text of Alice in Wonderland provided for reference. Images of the book covers and a carousel navigation allow readers to view the summaries.

Implications: Toward Scalable AI Alignment

By demonstrating effective summarization of long texts, the method illustrates a path for aligning AI systems on other complex, hard-to-evaluate tasks. The post suggests that scalable oversight techniques like this could improve the alignment of future general-purpose AI systems.