RUC-NLPIR/ARPO

[ICLR 2026] Agentic Reinforced Policy Optimization (ARPO)

What it solves

ARPO and AEPO address the instability and uncertainty introduced into Large Language Model (LLM) reasoning processes when interacting with external tools. Specifically, they tackle the challenge of aligning step-level tool-use behaviors in multi-turn agentic interactions, where the feedback from tool calls often results in high-entropy (uncertain) tokens that can hinder effective reinforcement learning.

How it works

ARPO (Agentic Reinforced Policy Optimization)

ARPO encourages the policy model to adaptively branch sampling during rounds where tool-call entropy is high. This allows the model to explore multiple potential reasoning paths more efficiently when it is most uncertain, aligning tool-use behaviors without requiring as many total tool calls as standard methods like GRPO.

AEPO (Agentic Entropy-Balanced Policy Optimization)

AEPO builds upon ARPO by balancing entropy during both the rollout and policy update phases:

  • Dynamic Entropy-Balanced Rollout: Uses entropy pre-monitoring to allocate sampling budgets and applies a branch penalty to prevent over-branching during consecutive high-entropy steps.
  • Entropy-Balanced Policy Optimization: Employs an "Entropy Clipping-Balanced Mechanism" (using stop-gradient operations) to rescale gradients on high-entropy tokens and uses entropy-aware advantage estimation to prioritize learning on tokens with high uncertainty.

Who it’s for

This project is designed for researchers and developers training LLM-based agents that require multi-turn tool interaction and complex reasoning capabilities.

Highlights

  • Improved Performance: Achieves high scores on benchmarks like GAIA and HLE (e.g., Qwen3-14B reaching 61.2% on GAIA).
  • Efficiency: Reduces the number of tool calls required during training compared to GRPO.
  • Optimization: Includes tool-call acceleration and memory optimizations, enabling the training of a 14B model on a single node with a batch size of 128 in just 10 minutes per step.
  • Comprehensive Resources: Provides a full codebase, SFT and RL datasets, and pre-trained model checkpoints ranging from 3B to 32B parameters.

Related

  • Project
    mll-lab-nu/RAGEN

    RAGEN is a reinforcement learning framework for training reasoning LLM agents over multiple interaction turns, built on the StarPO algorithm. It includes 10 built-in environments, supports PPO and GRPO optimization, and provides diagnostics for failure modes like the Echo Trap and reasoning collapse. RAGEN-2 adds SNR-Adaptive Filtering and mutual-information-based metrics to detect template collapse.

  • Project
    AgentR1/Agent-R1

    A modular framework for Agentic Reinforcement Learning that trains multi-step LLM agents by treating each interaction turn as a step-level MDP transition.

  • Project
    chi2liu/ABC-GRPO

    An asymmetric and adaptive refinement of the GRPO reinforcement learning algorithm that uses four independent clipping boundaries to prevent entropy collapse and improve generalization in LLM reasoning tasks.

  • Project
    areal-project/AReaL

    AReaL is a scalable, asynchronous reinforcement learning infrastructure designed to train reasoning and agentic models by decoupling training, inference, and agent execution into microservices.

  • Project
    AgileRL/AgileRL

    AgileRL is an open‑source Python library that unifies post‑training for large language models (SFT, DPO, RL algorithms) with classic deep‑RL (single‑agent, multi‑agent, offline, bandits). It offers YAML‑based experiment manifests, Torch‑native distributed training, LoRA/QLoRA adapters, evolutionary hyper‑parameter optimisation, and a managed cloud service called Arena for async, large‑scale runs. The same codebase can train LLMs locally or on GPU clusters, then deploy the fine‑tuned model for chat via a CLI.