RUC-NLPIR/ARPO

[ICLR 2026] Agentic Reinforced Policy Optimization (ARPO)

What it solves

ARPO and AEPO are reinforcement learning (RL) algorithms designed to improve how Large Language Model (LLM) agents handle multi-turn interactions and tool-use. They specifically address the high uncertainty (entropy) that occurs when an LLM receives feedback from external tools, which often disrupts the model's reasoning process and makes it difficult to align step-level behaviors.

How it works

ARPO (Agentic Reinforced Policy Optimization)

ARPO encourages the policy model to adaptively "branch sample" during tool-call rounds where entropy is high. This allows the model to explore multiple potential reasoning paths more effectively when it is most uncertain, leading to better alignment of tool-use behaviors.

AEPO (Agentic Entropy-Balanced Policy Optimization)

AEPO builds on these concepts by balancing entropy across two phases:

  • Rollout Phase: Uses a Dynamic Entropy-Balanced Rollout Mechanism to allocate sampling budgets and apply penalties to prevent over-branching during consecutive high-entropy steps.
  • Update Phase: Employs Entropy-Balanced Policy Optimization, which uses a stop-gradient operation (Entropy Clipping-Balanced Mechanism) to preserve gradients on high-entropy tokens and uses entropy-aware advantage estimation to prioritize learning on tokens with high uncertainty.

Who it’s for

This project is for AI researchers and developers training LLM-based agents (such as Qwen or Llama models) to perform complex reasoning and autonomous tool interaction in multi-turn environments.

Highlights

  • Improved Performance: Achieves high scores on benchmarks like GAIA and HLE (e.g., Qwen3-14B reaching 61.2% on GAIA).
  • Efficiency: Requires significantly fewer tool calls during training compared to GRPO.
  • Optimization: Includes tool-call acceleration and memory optimizations, enabling the training of a 14B model on a single node with a batch size of 128 in just 10 minutes per step.
  • Comprehensive Resources: Provides a full codebase, SFT and RL datasets, and pre-trained model checkpoints ranging from 3B to 32B parameters.

Related

  • Project
    jiangxinke/Harness-RL

    A reinforcement learning framework for multi-agent LLM systems that uses Conflict-Aware Policy Optimization to train agents without flattening execution sequences.

  • Project
    mll-lab-nu/RAGEN

    A flexible RL framework for training reasoning LLM agents using multi-turn reinforcement learning and the StarPO algorithm to prevent reasoning collapse.

  • Dispatch
    SAPO: A Stable and Performant Reinforcement Learning Method for Training Large Language Models

    Qwen introduces Soft Adaptive Policy Optimization (SAPO), an RL method that replaces hard clipping with a smooth, temperature-controlled gating function to improve training stability and sample efficiency in LLMs.

  • Project
    OpenPipe/ART

    An open-source RL framework that uses GRPO to train multi-step AI agents to learn from experience and improve reliability in real-world tasks.

  • Project
    AgentR1/Agent-R1

    A modular framework for Agentic Reinforcement Learning that trains multi-step LLM agents by treating each interaction turn as a step-level MDP transition.