agentscope-ai/Trinity-RFT

What it solves

Trinity-RFT provides a unified and flexible framework for Reinforcement Fine-Tuning (RFT) of Large Language Models. It simplifies the process of improving LLM capabilities through reinforcement learning by decoupling the experience generation, model updating, and data management into a coordinated system.

How it works

The framework is built around three core components:

• Explorer: Handles agent-environment interaction to generate experience data.
• Trainer: Updates the model weights by minimizing losses based on the collected data.
• Buffer: Manages the data pipeline, handling processing, cleaning, and augmentation throughout the RFT lifecycle.

It supports various RFT modes, including synchronous/asynchronous, on-policy/off-policy, and online/offline RL, and allows rollout and training to scale independently across devices.

Who it’s for

• Agent application developers looking to train LLM-powered agents for specific domains.
• Reinforcement learning researchers who want to implement and validate new RL algorithms using plug-and-play modules.
• Data engineers focused on creating RFT datasets and building complex data pipelines, including human-in-the-loop scenarios.

Highlights

• Agentic RL Support: Directly trains agent applications, including multi-step workflows and those developed with frameworks like AgentScope.
• Comprehensive Algorithm Support: Implements a wide array of algorithms including PPO, GRPO, DPO, SFT, and specialized variants like CHORD and REC.
• Full-Lifecycle Data Pipelines: Features active data management with prioritization, cleaning, and native support for multi-task joint learning.
• Flexible Deployment: Supports VLM training, LoRA, and a Tinker backend for users without local GPUs.