Liger GRPO and TRL Integration

Liger GRPO Integration with TRL

Liger GRPO is being integrated with the TRL (Transformer Reinforcement Learning) library to optimize Group Relative Policy Optimization (GRPO) training. However, current reports indicate compatibility issues when utilizing specific distributed training configurations.

DeepSpeed ZeRO-3 Compatibility Issue

A technical issue has been identified where Liger GRPO loss fails during multi-GPU training when configured with DeepSpeed ZeRO-3.

Error Details

When attempting to train the Qwen/Qwen2.5-0.5B-Instruct model using bf16 precision and DeepSpeed ZeRO-3, a torch._dynamo.exc.TorchRuntimeError occurs. The error is characterized as a size mismatch during a matrix multiplication (matmul) operation.

Specifically, the traceback reveals a shape mismatch in the following context:

• Input size: (s0x896)
• Vector size: (0)
• Error message: size mismatch, got input (s0x896), vec (0)

Technical Context

The failure occurs within the LigerFusedLinearGRPOFunction during the forward pass, specifically when calling accumulate_chunk in liger_kernel/chunked_loss/fused_linear_ppo.py. This suggests that the fused kernel may not be correctly handling the sharded parameters or the state management provided by DeepSpeed ZeRO-3, leading to an empty tensor (size 0) being passed to the matrix multiplication operation.