OpenAI Prover‑Verifier Games improve legibility of language model outputs
TL;DR
OpenAI introduced a prover‑verifier game training framework that teaches strong language models to generate solutions that weaker models can easily check, which also makes the outputs more legible for human evaluators.
Improving Legibility Through Optimization
Optimizing only for answer correctness can produce solutions that are hard for humans to follow. OpenAI showed that when human evaluators with limited time judged highly optimized solutions, they made almost twice as many errors as when evaluating less‑optimized solutions. By training a strong model (the prover) to produce text that a much weaker model (the verifier) can verify, the resulting outputs become both correct and easier for humans to assess. The experiments used grade‑school math problems and achieved about half the accuracy gain of pure correctness‑focused training while preserving human legibility.
How It Works
The training loop alternates between two phases:
- Verifier Training – A weak verifier model (≈3 orders of magnitude less pre‑training compute than the prover) learns to predict whether a prover‑generated solution is correct.
- Prover Training – The strong prover model is instructed to play either a helpful role (rewarding correct solutions that score highly on the verifier) or a sneaky role (rewarding incorrect solutions that nevertheless score highly on the verifier). Initially, the sneaky prover can fool the verifier, but each round adds those deceptive solutions to the verifier’s training set, eliminating the exploit. Training stops when the sneaky prover can no longer produce convincing incorrect answers.
This adversarial co‑training forces the prover to generate solutions that are transparent enough for the verifier—and consequently for humans—to evaluate.
Key Findings
- Performance vs. Legibility Trade‑off – Pure supervised fine‑tuning on human derivations yields high legibility but low accuracy; pure correctness optimization yields high accuracy but low legibility. The prover‑verifier game balances the two, retaining strong legibility with moderate accuracy.
- Three Useful Model Roles
- Robust Verifier – Accurately distinguishes correct from incorrect solutions, even when the prover attempts deception.
- Helpful Prover – Produces solutions that remain clear to human readers, reducing evaluator error rates.
- Sneaky Prover – Generates subtle, incorrect solutions that initially confuse evaluators, exposing weaknesses for further improvement.
The paper includes concrete examples (e.g., a multi‑step age‑calculation problem) showing how the helpful prover’s reasoning is explicit and easy to follow, whereas the sneaky prover’s reasoning contains deliberate mistakes that still receive high verifier scores early in training.
Implications for Alignment and Trustworthiness
Requiring language models to emit verifiable justifications reduces reliance on human judgments of legibility and moves toward autonomous alignment of future, more capable AI systems. Transparent, checkable outputs can increase trust in high‑stakes applications such as education, finance, and scientific research. Although the current study is limited to a single math dataset and still depends on ground‑truth labels, the methodology points to a scalable path for building AI systems whose correctness is evident without extensive human oversight.
Conclusion
The prover‑verifier game demonstrates that training strong models to produce text that weaker models can verify simultaneously improves human legibility and maintains a substantial portion of the accuracy boost from pure correctness optimization. This approach offers a promising route for aligning increasingly powerful language models with human values by making their reasoning processes transparent and easily auditable.
Authors: Yining Chen, Jan Hendrik Kirchner
Contributors: Angela Baek, Yuri Burda, Thomas Degry, Harri Edwards, Elie Georges, Cary Hudson, Jan Leike, Nat McAleese, Wes McCabe, Lindsay McCallum, Freddie Sulit