Qwen GSPO: Scalable Reinforcement Learning for Language Models

Qwen has introduced Group Sequence Policy Optimization (GSPO), a reinforcement learning (RL) algorithm designed to solve the instability and model collapse issues associated with scaling RL in large language models. By shifting from token-level to sequence-level optimization, GSPO enables continuous performance gains as training compute increases and simplifies the infrastructure required for Mixture-of-Experts (MoE) models.

Sequence-Level Optimization Objective

GSPO replaces token-level optimization with a sequence-level approach to improve stability and reduce variance. The algorithm defines an importance ratio based on the likelihood of the entire sequence rather than individual tokens.

To unify the numerical range and reduce variance, GSPO employs length normalization. The optimization objective is defined as follows:

$$\mathcal{J}{\text{GSPO}} (\theta) = \mathbb{E}{x \sim \mathcal{D}, {y_{i}}{i=1}^{G} \sim \pi{\theta_{old}} (\cdot \mid x)} \left[\frac{1}{G} \sum_{i=1}^{G} \min \left(s_{i}(\theta)(\hat{A}){i}, \text{clip}(s{i}(\theta), 1-\epsilon, 1+\epsilon)(\hat{A})_{i}\right)\right]$$

Where the sequence-level importance ratio $s_{i}(\theta)$ is calculated as:

$$s_{i}(\theta) = \left(\frac{\pi_{\theta}(y_{i} \mid x)}{\pi_{\theta_{old}}(y_{i} \mid x)}\right)^{\frac{1}{|y_{i}|}} = \exp \left(\frac{1}{|y_{i}|} \sum_{t=1}^{|y_{i}|} \log \frac{\pi_{\theta}(y_{i,t} \mid x, y_{i,<t})}{\pi_{\theta_{old}}(y_{i,t} \mid x, y_{i,<t})}\right)$$

Training Efficiency and Scalability

GSPO demonstrates higher training efficiency and better scalability compared to Group Relative Policy Optimization (GRPO). Experimental results using a cold-start model fine-tuned from Qwen3-30B-A3B-Base show that GSPO achieves superior performance on AIME'24, LiveCodeBench, and CodeForces benchmarks under the same training cost.

Key findings regarding scalability include:

  • Continuous Improvement: Performance continues to scale with increased training compute, regular updates to the query set, and extended generation lengths.
  • Learning Signal Reliability: Despite having a token clipping fraction two orders of magnitude higher than GRPO, GSPO achieves higher efficiency, suggesting that token-level optimization is noisier and less effective than the sequence-level approach.

Stability in Mixture-of-Experts (MoE) Models

GSPO inherently resolves stability challenges in the RL training of large MoE models, specifically eliminating the need for "Routing Replay."

In GRPO, expert activation volatility often prevents convergence, requiring a Routing Replay strategy that caches activated experts from the old policy ($\pi_{\theta_{old}}$) and replays them in the current policy ($\pi_{\theta}$). This workaround adds memory and communication overhead and can limit model capacity.

Because GSPO focuses on sequence-level likelihood ($\pi_{\theta}(y_{i} \mid x)$) and is not sensitive to individual token likelihoods, it removes the dependency on Routing Replay, simplifying the training process and allowing MoE models to maximize their capacity.

Infrastructure and Precision Benefits

GSPO's sequence-level optimization makes the training process more tolerant of precision discrepancies. This allows the use of likelihoods returned directly from inference engines for optimization, removing the requirement to recompute them within training engines. This capability is particularly advantageous for:

  • Partial rollouts
  • Multi-turn RL
  • Training-inference disaggregated frameworks

These technical advancements in GSPO served as the algorithmic foundation for the performance improvements seen in the latest Qwen3 (Instruct, Coder, and Thinking) models.

Sources