TsinghuaC3I/MARTI

A Framework for LLM-based Multi-Agent Reinforced Training and Inference

What it solves

MARTI addresses the difficulty of training Large Language Model (LLM) based multi-agent systems (MAS) using reinforcement learning (RL). It provides a scalable way to manage complex multi-agent interactions and reward allocation while distributing the actual policy training across individual agents. Its latest version, MARTI-v2, specifically targets complex reasoning tasks like code generation where standard token-level optimization often falls short.

How it works

The framework operates on a principle of centralized interaction and distributed training. It consists of three core modules: a Multi-Agent World for interactions, Centralized Rewarding for credit assignment, and Single Agent Trainers for policy updates.

MARTI-v2 introduces MARS² (Multi-Agent Tree Search RL), which uses asynchronous multi-agent tree search to explore solution spaces more systematically through adaptive node expansion and refinement. To stabilize training for long sequences (up to 32K tokens), it employs GSPO loss for sequence-level optimization and Truncated Importance Sampling (TIS) to correct sampling biases from the vLLM engine.

Who it’s for

AI researchers and developers building collaborative LLM agent teams for complex reasoning, mathematics, or code generation tasks who need a robust infrastructure for reinforcement learning.

Highlights

  • Multi-Agent Tree Search: Enables efficient exploration of reasoning trajectories for high-quality solution discovery.
  • Heterogeneous Training: Supports training different models (e.g., Qwen3 and AreaL) simultaneously with independent roles and strategies.
  • Diverse RL Algorithm Support: Compatible with PPO, GRPO, REINFORCE++, and TTRL.
  • Flexible Workflows: Built-in support for graph-based workflows like Multi-Agent Debate, Chain-of-Agents, and Mixture-of-Agents.
  • Infrastructure Integration: Built on OpenRLHF and supports vLLM for fast inference and training.

Related

  • Project
    OpenManus/OpenManus-RL

    An open-source framework for optimizing LLM agents using reinforcement learning to enhance their reasoning, tool use, and decision-making capabilities.

  • Project
    AgentR1/Agent-R1

    A modular framework for Agentic Reinforcement Learning that trains multi-step LLM agents by treating each interaction turn as a step-level MDP transition.

  • Project
    agentscope-ai/Trinity-RFT

    A general-purpose framework for reinforcement fine-tuning of LLMs that decouples exploration, training, and data buffering to support diverse RL algorithms and agentic workflows.

  • Project
    mll-lab-nu/VAGEN

    A reinforcement learning framework that trains multi-turn VLM agents to build internal world models by reinforcing explicit visual state reasoning and transition modeling.