Train Time Scaling and Scaling Reinforcement Learning for Self‑Improving AI Agents

Train‑Time Scaling Overview

The core idea of train‑time scaling is to use a model’s own outputs, filter them by correctness, and fine‑tune the model on those filtered outputs, thereby substituting compute for model parameters. This creates a loop where generated solutions are verified, kept if correct, and used to improve the model further. The loop works best in domains with strong verifiers, such as mathematical reasoning or code execution, where the final answer can be checked automatically.

STaR: Bootstrapping Rationales

STaR (Self‑Taught Reasoner) bootstraps reasoning chains by starting with a small set of question‑rationale‑answer examples, generating answers for a larger set of questions, keeping only those with correct answers, and for the incorrect attempts giving the correct answer as a hint and asking the model to produce a rationale. The model is then fine‑tuned on the collected rationales and answers, and the process can be repeated iteratively. As the transcript states, "the key insight that STaR has is very simple"—it treats correctness of the final answer as a proxy for reasoning quality and assumes the model can generate valid rationales when given the answer as a hint. Experiments on GPT‑J (6 B) showed improved performance on CommonsenseQA (reaching 72.5 % accuracy while using only 86 % of the data needed for standard supervised fine‑tuning) but little gain on GSM8K, where direct fine‑tuning on good data performed similarly. STaR plateaus after several iterations because it is not a full reinforcement learning method.

DeepSeekMath and GRPO

DeepSeekMath scales up reinforcement learning for mathematical reasoning by starting from a code‑pre‑trained model, curating math data from Common Crawl (OpenWebMath), and applying Group Relative Policy Optimization (GRPO). GRPO replaces the critic and value function used in PPO with a group‑based advantage estimate: for each question, many answers are sampled, scored by a reward model, and the advantage is computed as (reward − mean reward) / stddev reward. This requires only three model copies instead of the four needed for PPO, making it memory‑efficient. On a simpler math benchmark (not AIME), DeepSeekMath raised accuracy from 46.8 % to 51.7 % using a 7 B model. The improvement was seen in majority‑at‑K metrics, while pass‑at‑K did not increase, indicating greater consistency rather than a jump in fundamental problem‑solving ability.

DAPO: Stabilizing RL for Long Reasoning

DAPO addresses three instability issues that arise when applying GRPO to long chain‑of‑thought reasoning: entropy collapse, uncontrolled response length, and noisy truncated outputs. It introduces asymmetric clipping (allowing larger increases than decreases in the PPO clip), dynamic sampling (filtering out groups with all‑zero or all‑one rewards to keep a useful gradient signal), token‑level loss (weighting loss by token count instead of treating each sequence equally), and a gradual penalty for overlong tokens to discourage uncontrolled generation. Applied to the AIME benchmark with a Qwen‑32B model, the baseline GRPO gave ~30 % accuracy; adding overlong filtering raised it to 36 %, asymmetric clipping to 38 %, soft overlong punishment to 41 %, token‑level loss to 42 %, and dynamic sampling to ~50 %. As with the prior methods, the gain was in majority‑at‑K performance, not pass‑at‑K.

Results on AIME and Reasoning Benchmarks

The lecture contrasts the traditional scaling‑with‑parameters view with the train‑time scaling results: a 175 B parameter model (GPT‑3.5) scores only ~5 % on AIME, whereas a 7 B model trained with DeepSeekMath reaches 51.7 % (up to 60 % with extra tricks) and a 32B model trained with DAPO reaches ~50 %. These results show that investing compute in generating and filtering reasoning traces can substitute for model size, but the primary effect is improved consistency across multiple samples (higher majority‑at‑K) rather than the ability to solve previously unsolvable problems (pass‑at‑K).

Open Questions and Limitations

The talk ends with several open research directions: why majority‑at‑K improves while pass‑at‑K does not; how to learn from failed attempts instead of filtering them out; how to obtain sufficient verification signals when data are scarce (e.g., using ensembles of verifiers); and whether combining STaR‑style rationalization with DAPO‑style RL stabilization can yield further gains. It also notes that train‑time scaling still relies on a strong reward signal and verifier, and that the approach does not yet teach models to generalize to entirely new problem types beyond improving consistency on known formats.