opendilab/LightRFT

What it solves

LightRFT provides a high-performance framework for Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from Verifiable Rewards (RLVR). It addresses the complexity and resource intensity of fine-tuning Large Language Models (LLMs) and Vision-Language Models (VLMs) by offering efficient inference engines, diverse optimization algorithms, and advanced memory management to reduce GPU overhead.

How it works

LightRFT integrates high-speed inference backends like vLLM and SGLang to handle sampling and generation. It employs a variety of policy optimization algorithms (such as GRPO and GSPO) and advantage estimation techniques to refine model behavior. To maximize hardware efficiency, it uses a "Colocate Anything" approach, allowing reward models and training models to share the same GPU device, and supports distributed training strategies like FSDP v2 and DeepSpeed ZeRO.

Who it’s for

It is designed for AI researchers and developers working on alignment and reinforcement learning for multimodal models, specifically those looking to implement complex RLHF/RLVR pipelines for text, vision-language, and audio-language tasks.

Highlights

• Omni-modal Support: Native training pipelines for LLMs, VLMs (e.g., Qwen-VL), and audio-language models.
• Diverse Algorithm Suite: Implements state-of-the-art methods including GRPO, GSPO, DAPO, and REINFORCE++.
• Resource Optimization: Features FP8 inference, engine sleep/wake mechanisms, and co-location of reward models to minimize communication overhead.
• Scalable Infrastructure: Built-in support for FSDP v2, DeepSpeed ZeRO (Stages 1-3), and mixed precision training (BF16/FP16).
• Integrated Tooling: Includes Weights & Biases integration and benchmarking tools for math capabilities (GSM8K, Geo3K).