SAPO: A Stable and Performant Reinforcement Learning Method for Training Large Language Models

TL;DR

Qwen has introduced Soft Adaptive Policy Optimization (SAPO), a reinforcement learning (RL) method designed to stabilize the training of large language models (LLMs). By replacing the hard clipping mechanisms used in existing methods like GRPO and GSPO with a smooth, temperature-controlled gating function, SAPO prevents unstable policy updates while preserving more useful gradient information, leading to higher performance in reasoning, coding, and multimodal tasks.

The Challenge of Policy Optimization in LLMs

Stable policy optimization is difficult in LLMs, particularly for large Mixture-of-Experts (MoE) models, due to the variance of token-level importance ratios. These ratios measure the deviation of the current policy from the behavior policy used to generate training samples.

Existing group-based policy optimization methods attempt to control this instability through hard clipping:

  • GRPO (token-level clipping): Truncates gradients whenever the importance ratio falls outside a fixed band.
  • GSPO (sequence-level clipping): Employs clipping at the sequence level.

These hard-clipping approaches suffer from two primary limitations: the loss of learning signals (where informative samples are discarded) and a brittle trade-off between stability and sample efficiency. If the clipping range is too tight, useful gradients are lost; if it is too wide, noisy gradients destabilize the model.

Soft Adaptive Policy Optimization (SAPO) Explained

SAPO addresses these limitations by replacing hard clipping with a smooth gating function $f_{i,t}(x) = \frac{4}{\tau_{i,t}} \cdot \sigma\big(\tau_{i,t}(x - 1)\big)$. This function adaptively down-weights off-policy updates instead of abruptly cutting them off.

Key Technical Features

  • Continuous Trust Regions: SAPO avoids the discontinuities associated with hard clipping, providing a smooth decay of contributions as the policy deviates.
  • Sequence-Level Coherence: SAPO maintains the sequence-level behavior seen in GSPO. However, unlike GSPO, which suppresses an entire sequence if a few tokens are off-policy, SAPO only suppresses the problematic tokens, thereby preserving other useful gradients and improving sample efficiency.
  • Token-Level Adaptivity: The method allows for selective suppression of tokens that are too far off-policy, preventing unstable policy shifts without discarding the entire learning signal.
  • Asymmetric Temperature Design: SAPO utilizes different temperatures for positive and negative advantages ($\tau_{\text{neg}} > \tau_{ ext{pos}}$). Because negative advantages can increase the logits of many inappropriate tokens in large vocabularies, a higher temperature for negative tokens causes those contributions to decay faster when off-policy, significantly improving training stability.

Experimental Results

Qwen tested SAPO across various model architectures, including both dense and MoE models.

Mathematical Reasoning (Qwen3-30B-A3B)

Using a cold-start model fine-tuned from Qwen3-30B-A3B-Base, SAPO was compared against GSPO and GRPO-R2 (GRPO with routing replay). The results showed:

  • SAPO maintained stable training for longer periods than GSPO and GRPO-R2.
  • SAPO achieved higher final Pass@1 scores on AIME25, HMMT25, and BeyondAIME benchmarks.
  • SAPO eliminates the need for routing replay, simplifying the RL pipeline.

Large-Scale RL for Qwen3-VL Models

SAPO was applied to Qwen3-VL-30B-A3B on a mixture of math, coding, logic, and multimodal tasks. Evaluation benchmarks included AIME25, LiveCodeBench v6, ZebraLogic, and MathVision. Findings indicated that SAPO consistently outperformed both GSPO and GRPO-R2 under the same compute budget across varying model sizes and both MoE and dense architectures.

Implications for LLM Training

SAPO provides a practical framework for enhancing RL training by ensuring that updates remain aligned with sequence-level behavior while maintaining token-level flexibility. By solving the brittleness of hard clipping and implementing asymmetric temperature control, SAPO reduces the impact of high-variance updates, making it a more stable and efficient foundational component for future RL-trained LLMs.

Sources