AgentR1/Agent-R1

Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning

What it solves

Agent-R1 addresses the difficulty of training multi-step LLM agents using reinforcement learning (RL). Traditional RL pipelines often treat entire interactions as a single long prompt-response sequence, which makes it hard to manage tool use, environment states, and precise reward assignment for specific actions within a multi-turn conversation.

How it works

The framework implements a "Step-level MDP" (Markov Decision Process) approach. Instead of treating the interaction as one long string, it treats each individual agent step—observing the environment, taking an action, and receiving feedback—as the basic unit of training.

It uses a layered architecture to decouple different components:

  • AgentFlow: Manages prompt construction and context.
  • AgentEnvLoop: Connects the model to an environment's reset/step interface.
  • AgentEnv/ToolEnv: Defines the logic of the task and the tools available.
  • BaseTool: A standard interface for creating executable tools (e.g., calculators or APIs).

This structure allows the model to undergo a loop of rollout, reward calculation, replay, and policy update while maintaining clear boundaries between actions and observations.

Who it’s for

It is designed for researchers and developers building autonomous LLM agents that require multi-step reasoning, tool interaction, and optimization via reinforcement learning.

Highlights

  • Step-native RL: Models every turn as a step-level transition for better credit assignment.
  • Flexible Context: Allows the environment to determine how history is appended, truncated, or summarized.
  • Algorithm Decoupling: Supports various RL algorithms (like GRPO, PPO, and REINFORCE) independently of the task workflow.
  • Modular Design: Provides layered abstractions so new tasks can be added without rewriting the entire RL stack.

Related

  • Project
    OpenPipe/ART

    An open-source RL framework that uses GRPO to train multi-step AI agents to learn from experience and improve reliability in real-world tasks.

  • Project
    langfengQ/verl-agent

    An extension of veRL for training LLM agents via reinforcement learning, featuring a step-independent rollout mechanism to efficiently handle long-horizon, multi-turn tasks.

  • Project
    mll-lab-nu/VAGEN

    VAGEN is a reinforcement learning framework for training multi-turn VLM agents that improves performance by reinforcing their internal world model through explicit visual state reasoning.

  • Project
    TsinghuaC3I/MARTI

    An open-source framework for training LLM-based multi-agent systems using reinforcement learning, featuring tree search augmentation for complex reasoning and code generation.

  • Project
    Osilly/Vision-DeepResearch

    A framework and benchmark for developing Multimodal Large Language Models capable of deep research through iterative visual and textual search across images and videos.