chi2liu/ABC-GRPO

What it solves

ABC-GRPO addresses the instability and generalization issues found in standard Group Relative Policy Optimization (GRPO). Specifically, it fixes a "blind spot" in GRPO's clipping mechanism where certain token updates (those with negative advantages and high likelihood ratios) are unbounded, leading to excessive punishment, entropy collapse, and a decline in the model's ability to actually solve problems across multiple attempts (Pass@k degradation).

How it works

Instead of using two conditional clipping boundaries that depend on the sign of the advantage, ABC-GRPO introduces four independent clipping boundaries ($\epsilon_1, \epsilon_2, \epsilon_3, \epsilon_4$). This ensures that gradient updates are bounded across all four quadrants of the ratio-advantage space, regardless of whether the advantage is positive or negative. By providing unconditional bounds, the algorithm prevents the model from over-suppressing correct tokens in failed sequences, which preserves higher entropy and maintains the model's exploration capacity during training.

Who it’s for

Researchers and developers training Large Language Models (LLMs) using reinforcement learning for reasoning tasks (such as mathematics), who want to improve training stability and prevent the model from converging prematurely to a narrow set of answers.

Highlights

• 4-Boundary Clipping: Replaces conditional clipping with independent parameters for all four quadrants.
• Entropy Preservation: Maintains significantly higher entropy (up to 10.9$×$ higher than GRPO), reducing the risk of entropy collapse.
• Improved Generalization: Demonstrates monotonic improvement in Pass@k metrics on mathematical benchmarks (AIME, AMC) where standard GRPO often degrades.
• Bounded Updates: Eliminates unbounded gradient updates in the Q2 and Q4 quadrants, which account for nearly half of all clipping events.