OpenAI Improving Mathematical Reasoning with Process Supervision
OpenAI has achieved a new state-of-the-art in mathematical problem solving by implementing process supervision, a method that rewards a model for every correct step of reasoning instead of only rewarding the correct final answer. This approach not only boosts mathematical performance but also improves AI alignment by training models to produce human-endorsed chains-of-thought.
Process Supervision vs. Outcome Supervision
Process supervision provides feedback for each individual step in a chain-of-thought, whereas outcome supervision provides feedback based solely on the final result. This distinction is critical for mitigating "hallucinations"—logical mistakes that occur even in state-of-the-art large language models.
By rewarding the process, the model is directly trained to follow a logical path that is endorsed by humans. In contrast, outcome supervision may inadvertently reward an unaligned or incorrect reasoning process that happens to arrive at the correct final answer, making it harder to scrutinize and less reliable.
Impact on AI Alignment and the "Alignment Tax"
Process supervision offers several distinct advantages for AI alignment:
- Interpretable Reasoning: Because the model is encouraged to follow a human-approved process, the resulting reasoning chains are more interpretable.
- Direct Alignment: Each step of the reasoning process receives precise supervision, ensuring the model adheres to an aligned chain-of-thought.
- Negative Alignment Tax: Typically, implementing safer AI methods can lead to a reduction in performance, a phenomenon known as the "alignment tax." However, in the domain of mathematics, process supervision incurs a "negative alignment tax," meaning it actually increases the model's capability while simultaneously improving alignment.
Performance Benchmarks on the MATH Dataset
OpenAI evaluated process-supervised and outcome-supervised reward models using the MATH test set. The evaluation involved generating multiple solutions for each problem and selecting the solution ranked highest by the respective reward model.
Key findings include:
- Superior Accuracy: The process-supervised reward model outperformed the outcome-supervised model across all tested scenarios.
- Increased Reliability: The performance gap between the two methods widened as the number of solutions considered per problem increased, indicating that the process-supervised reward model is more reliable at identifying the correct reasoning path.
Case Study: Complex Trigonometry
To demonstrate the effectiveness of the reward model, OpenAI highlighted a challenging trigonometry problem involving the simplification of $\tan 100^{\circ} + 4 \sin 100^{\circ}$.
While GPT-4 typically struggles with this specific problem—with only 0.1% of solution attempts reaching the correct answer—the process-supervised reward model was able to correctly recognize a valid solution. The valid solution required a complex, 26-step sequence of trigonometric identities and simplifications to arrive at the correct answer of $-\sqrt{3}$.
Future Research Directions
While these results are significant for mathematical reasoning, OpenAI notes that it remains unknown how broadly process supervision will generalize to other domains. Future research will focus on exploring whether this method can provide a similar combination of increased performance and improved alignment in non-mathematical tasks.