hiyouga/EasyR1

EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework based on veRL

What it solves

EasyR1 is a training framework designed to make reinforcement learning (RL) for large language models (LLMs) and vision-language models (VLMs) more efficient and scalable. It specifically addresses the complexity of training reasoning-capable models (like DeepSeek-R1 distillations) across different modalities.

How it works

It is a fork of the veRL project that integrates the HybridEngine and vLLM's SPMD mode to optimize performance. The framework supports a wide range of RL algorithms, including GRPO, DAPO, Reinforce++, ReMax, RLOO, GSPO, and CISPO. It allows for both full fine-tuning and LoRA training, and it can scale to multi-node environments using Ray.

Who it’s for

It is intended for AI researchers and developers who need to train multimodal reasoning models using reinforcement learning, particularly those who are working with Llama, Qwen, and DeepSeek architectures.

Highlights

  • Multimodal Support: Supports both text-only and vision-language models (e.g., Qwen2-VL).
  • Algorithm Variety: Implements multiple RL algorithms including GRPO and DAPO.
  • Scalability: Supports multi-node training via Ray and padding-free training for efficiency.
  • Flexible Training: Offers both full fine-tuning and LoRA options.
  • Broad Model Compatibility: Works with Llama3, Qwen2/2.5/3, and DeepSeek-R1 distill models.

Related

  • Project
    om-ai-lab/VLM-R1

    VLM-R1 is a framework for training R1-style Large Vision-Language Models using GRPO to improve reasoning and generalization in tasks like visual grounding, math, and object detection.

  • Project
    2U1/Qwen-VL-Series-Finetune

    A comprehensive fine-tuning toolkit for the Qwen-VL series of vision-language models, supporting SFT, DPO, GRPO, and reasoning-mode training for images and videos.

  • Project
    verl-project/verl-omni

    A general RL training framework designed for multimodal generative models, providing fast and stable post-training for diffusion and omni-modality models.

  • Project
    opendilab/LightRFT

    A high-performance reinforcement learning fine-tuning framework for LLMs and VLMs that supports RLHF and RLVR with efficient inference engines and multimodal capabilities.

  • Project
    PrimeIntellect-ai/prime-rl

    A framework for large-scale asynchronous reinforcement learning designed to train 1T+ parameter MoE models across 1000+ GPUs using FSDP2 and vLLM.