jiangxinke/Harness-RL

Agentic RAG R1 Framework via Reinforcement Learning

What it solves

Harness-RL addresses the challenge of training language-model agents within a multi-agent system (harness) without needing to flatten the entire execution process into a single, long token sequence. It specifically solves the problem of attributing credit (gradients) correctly when an agent's action and the arguments for that action are decoupled, allowing for more efficient reinforcement learning in complex, multi-agent environments.

How it works

The system constructs "black-box trajectories" from interface calls and session prefix trees. To handle the decoupling of actions and arguments, it uses Conflict-Aware Policy Optimization (CAPO). This technique routes gradients from action tokens and argument tokens to their respective parameter subspaces, ensuring that the model is updated based on the accuracy of the specific component (the action or the argument) rather than a generic sequence update.

Who it’s for

This project is designed for AI researchers and developers working on reinforcement learning for LLM-based agents, particularly those building central-agent architectures that coordinate multiple specialist agents.

Highlights

  • Action-Args Decoupling: Prevents the need to flatten execution sequences, maintaining the structure of the multi-agent interaction.
  • Conflict-Aware Policy Optimization (CAPO): A specialized gradient routing mechanism to optimize parameter subspaces.
  • Flexible Training Modes: Supports both training only the central agent and joint training across all agents in the harness.
  • Integrated Infrastructure: Includes slime for distributed rollout, SGLang serving, and Megatron training.

Related

  • Project
    RUC-NLPIR/ARPO

    A suite of agentic reinforcement learning algorithms (ARPO and AEPO) that optimize LLM tool-use and reasoning by adaptively managing entropy during multi-turn interactions.

  • Project
    mll-lab-nu/RAGEN

    A flexible RL framework for training reasoning LLM agents using multi-turn reinforcement learning and the StarPO algorithm to prevent reasoning collapse.

  • Project
    OpenManus/OpenManus-RL

    An open-source framework for optimizing LLM agents using reinforcement learning to enhance their reasoning, tool use, and decision-making capabilities.

  • Project
    OpenPipe/ART

    An open-source RL framework that uses GRPO to train multi-step AI agents to learn from experience and improve reliability in real-world tasks.

  • Project
    om-ai-lab/VLM-R1

    VLM-R1 is a framework for training R1-style Large Vision-Language Models using GRPO to improve reasoning and generalization in tasks like visual grounding, math, and object detection.