AI-written critiques help humans notice flaws

OpenAI has developed "critique-writing" models that describe flaws in summaries, enabling human evaluators to identify errors more effectively than they would alone. This research serves as a proof of concept for using AI assistants to help humans supervise AI systems on tasks that are too complex for humans to evaluate manually.

AI Assistance Increases Human Error Detection

AI-generated critiques significantly improve the ability of humans to spot flaws in text summaries. In experiments using topic-based summaries of short stories, Wikipedia articles, and other internet texts, OpenAI found that human labelers assisted by model-written critiques identified 50% more flaws in model-written summaries compared to a control group receiving no assistance.

For summaries that were deliberately written by humans to contain subtle flaws, the assistance increased the human detection rate of intended flaws from 27% to 45%.

Scaling Properties of Critique Writing

Model scale correlates positively with the ability to generate helpful self-critiques and use those critiques for self-improvement.

  • Self-Critique Capability: Larger models are more proficient at self-critiquing than smaller models.
  • Iterative Improvement: Large models can use their own self-critiques to directly improve their outputs, a capability that smaller models lack. The quality of the improvement is tied to the quality of the critique; better critiques lead to better improvements.
  • Scaling Impact: Scale improves the ability to write critiques more than it improves the ability to write the original summaries.

The Gap Between Discrimination and Articulation

There is a measurable gap between a model's ability to detect a flaw (discrimination) and its ability to explain that flaw (critique).

OpenAI researchers found that models are better at discriminating whether an answer is flawed than they are at producing a concrete critique that a human can understand. This gap between discrimination and critique ability does not appear to decrease as models get larger. This finding is critical for alignment research, as it suggests models may "know" about problems they cannot or will not articulate to human supervisors.

Implications for AI Alignment

Ensuring that AI systems remain aligned with human intent is difficult when humans cannot easily spot errors in complex outputs, such as bugs in a codebase or factual errors in long essays. If humans cannot detect these errors, models may learn to produce outputs that look correct but contain systematic flaws.

AI-assisted evaluation is a key pillar of OpenAI's alignment strategy. The goal is to develop assistants that can handle the cognitive labor of evaluation, allowing humans to focus on communicating their preferences. This work builds upon other alignment research, including debate and recursive reward modeling.

Sources