Self-Improving AI Agents Verification Methods Overview

Training Verifiers to Solve Math Word Problems (OpenAI 2021)

The core takeaway is that a verifier trained to predict the correctness of a full solution can improve answer selection when combined with repeated sampling, especially as the verifier training set grows. The paper introduced the GSM8K dataset of 8,500 grade‑school math problems that require multi‑step reasoning and natural‑language solutions. To train the verifier, the authors fine‑tuned a language model for two epochs on a portion of GSM8K, then generated 100 completions per problem, labeled each completion as correct or incorrect using human‑provided final answers, and trained a verifier for one epoch on this labeled data. The verifier architecture is a language model with a scalar head that outputs a binary prediction per token; at test time the score of the final token is used to rank completions and pick the highest‑scoring one. Ablation studies showed that verification outperforms pure supervised fine‑tuning when the verifier sees more than about 1,000 labeled examples, and that a larger generator paired with a smaller verifier tends to work better than the reverse. Increasing the number of completions per problem improves accuracy up to around 400 samples; beyond that the verifier’s precision drops because it cannot distinguish closely competing correct and incorrect answers.

Let's Verify Step by Step (OpenAI)

The main conclusion is that process‑supervised reward models (PRMs) provide finer‑grained credit assignment and are more robust to hallucination than outcome‑supervised reward models (ORMs), especially when human‑labeled step‑wise correctness is available. The authors created the PRM800K dataset containing 800,000 human‑annotated step labels for model‑generated reasoning chains. For ORM training they used the final token’s score as the reward; for PRM training they multiplied the per‑step probabilities to obtain a sequence‑level reward. Experiments on GSM8K showed that PRM outperforms ORM and majority voting, correctly identifying rare correct solutions that ORM misses, and PRM benefits more from additional labels than ORM does. Process supervision reduces false positives because a model cannot obtain a high reward by arriving at a correct final answer through an incorrect reasoning path; the step‑wise labels penalize such shortcuts. The authors note that combining PRM and ORM signals can capture the benefits of both, but introduces a threshold hyperparameter that must be tuned.

Math‑Shepherd: Automatic Step‑Level Annotation

The key result is that step‑wise labels can be generated automatically without human annotation by estimating each step’s potential to reach a correct final answer through sampling, and that this automated PRM can be used as a reward model for reinforcement learning to further improve the generator. Given a current reasoning step, the method samples N continuations; a hard estimate labels the step as successful if any continuation reaches a correct final answer, while a soft estimate uses the fraction of successful continuations. The authors found that with N = 4 the hard and soft estimates give similar performance, and they chose the hard estimate for simplicity. Using these automatically generated labels, they trained a PRM and then applied PPO to fine‑tune the generator (e.g., Mistral‑7B) against the PRM, achieving higher accuracy on GSM8K and a larger gain on the more challenging MATH dataset compared to using an ORM‑based reward. The approach also works with self‑consistency (majority voting) baselines, showing that the learned PRM provides a stronger signal than simple vote‑based selection.

Weaver: Ensemble of Weak Verifiers

The central finding is that combining many imperfect verifiers through weak‑to‑strong supervision (e.g., Naive Bayes or logistic regression) yields a significantly stronger verifier, and that the resulting ensemble can be distilled into a small model that retains most of the accuracy while using far less test‑time compute. Weaver treats each verifier (PRM, ORM, LLM judge, etc.) as providing a noisy label for a candidate solution; under the assumption that verifier errors are independent, it estimates verifier accuracies from a small labeled set and computes optimal weights to combine their scores. Low‑quality verifiers are filtered out before weighting. On hard benchmarks such as GPQA Diamond, MATH, and MLU Pro, the weighted ensemble improves accuracy from just over 40% to over 70% for an 8B‑parameter generator, matching the performance of much larger models like o3‑mini; scaling to a 70B generator pushes average accuracy to 86.2%. Distilling the weighted ensemble into a 400‑million‑parameter model preserves 97% of the ensemble’s accuracy while reducing test‑time compute by 99% or more. All checkpoints are released open‑source for use in agentic or test‑time scaling projects.