AgentR1/Agent-R1
Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning
What it solves
Agent-R1 addresses the gap between LLM serving systems and distributed training systems in the context of multi-turn agent interactions. It replaces single-turn RL pipelines—which treat interactions as one long prompt-response sequence—with a framework that treats every turn as a distinct step-level Markov Decision Process (MDP) transition. This allows for more explicit and precise management of tool use, environment state, and reward assignment.
How it works
The framework operates on a rollout -> reward -> replay -> update loop. It treats the "agent step" as the basic interaction unit, recording the observation, action, produced feedback, and reward for each turn.
It uses a layered architecture to decouple the training stack from the task logic:
- AgentFlowBase: Manages prompt construction and model calls.
- AgentEnvLoop: Connects model generation to an environment's reset/step interface.
- AgentEnv/ToolEnv: Defines the environment logic and tool registration.
- BaseTool: The standard interface for executable tools (e.g., calculators or APIs).
Who it’s for
AI researchers and developers building multi-step LLM agents that require reinforcement learning to improve their decision-making and tool-use capabilities across various environments (e.g., academic paper search, table reasoning, or web shopping).
Highlights
- Step-level MDP representation: Avoids fragile text reconstruction by storing transitions as structured data.
- Flexible context management: Allows the environment to truncate, summarize, or rewrite history based on the the task needs.
- Algorithm-system decoupling: Enables independent evolution of rollout, rewards, and policy objectives.
- Broad algorithm support: Compatible with GRPO, PPO, REINFORCE, and RLOO, as well as StepPO.