mll-lab-nu/RAGEN

What it solves

RAGEN addresses the challenge of training LLM agents to reason through multi-turn interactions with an environment. It specifically targets failure modes in agentic reinforcement learning (RL), such as the "Echo Trap" and "reasoning collapse," where models produce reasoning that looks diverse but is actually input-agnostic and disconnected from the actual state of the task.

How it works

Built on the StarPO (State-Thinking-Actions-Reward Policy Optimization) framework, RAGEN uses a two-stage process of rollouts and updates. In the rollout stage, the LLM generates reasoning-guided actions (formatted as <think>...</think><ans> action </ans>) and receives environment feedback. In the update stage, the framework optimizes entire trajectories using importance sampling via PPO or GRPO.

RAGEN-2 introduces SNR-Adaptive Filtering, which uses reward variance as a proxy to filter out noisy rollout data, and mutual information proxy metrics to diagnose and monitor "template collapse" (where reasoning becomes a generic template regardless of the input).

Who it’s for

It is designed for researchers and developers training reasoning-capable LLM agents for complex, multi-turn tasks across various environments.

Highlights

• StarPO Framework: A unified RL framework supporting both trajectory-level and turn-wise training.
• Built-in Environments: Includes 10 pre-configured environments like Sokoban, WebShop, and DeepCoder.
• Gym-Compatible: Easy integration of custom environments.
• Reasoning Diagnostics: Advanced metrics (Conditional Entropy and Mutual Information) to detect reasoning collapse.
• SNR-Adaptive Filtering: A lightweight method to stabilize training and mitigate noisy gradient updates.