hiyouga/EasyR1

EasyR1 – Efficient, Scalable Multi‑Modality RL Training

What it is

• A fork of the veRL reinforcement‑learning‑from‑human‑feedback (RLHF) framework, extended to handle vision‑language models (VLMs).
• Built around the HybridEngine (from the HybridFlow paper) and the SPMD mode of vLLM, giving fast, memory‑efficient training on large LLMs and VLMs.

Key capabilities

• Model support: Llama 3, Qwen 2/2.5/3 (text) and their vision‑augmented variants, plus DeepSeek‑R1 distilled models.
• RL algorithms: GRPO, DAPO, Reinforce++, ReMax, RLOO, GSPO, CISPO (many are recent RL‑HF methods).
• Training tricks: padding‑free batches, LoRA‑style low‑rank adaptation, checkpoint resume, and integration with popular experiment trackers (Wandb, SwanLab, Mlflow, TensorBoard).
• Scalability: works from a single GPU up to 70‑B‑parameter models across many GPUs or nodes via Ray and FSDP, with hardware‑usage tables for AMP/BF16.
• Dataset flexibility: any text or image‑text dataset that follows the provided format; examples include Math12K, Geometry3K, JourneyBench‑multi‑image‑VQA, and a mixed RL dataset.

Typical workflow

1. Install – clone the repo and run pip install -e . or use the provided Docker image (hiyouga/verl).
2. Prepare data – place your text or vision‑text data in the expected JSON/Parquet layout.
3. Run a script – e.g., bash examples/qwen2_5_vl_7b_geo3k_grpo.sh for full‑parameter GRPO training, or the LoRA variant script for parameter‑efficient fine‑tuning.
4. Merge checkpoints – python scripts/model_merger.py converts the RL‑trained checkpoint back to a Hugging‑Face‑compatible model.
5. (Optional) Multi‑node – start a Ray cluster and launch the same script on the head node.

Getting started quickly

git clone https://github.com/hiyouga/EasyR1.git
cd EasyR1
pip install -e .
# Full‑parameter training on a 7B Qwen2.5‑VL model
bash examples/qwen2_5_vl_7b_geo3k_grpo.sh

The repo also includes a LoRA script, SwanLab logging example, and a Dockerfile for reproducible environments.

Who might use it

• Researchers fine‑tuning multimodal LLMs with reinforcement learning to improve reasoning, tool use, or visual question answering.
• Engineers needing a production‑ready RLHF pipeline that scales from a single GPU to multi‑node clusters.
• Anyone wanting to experiment with newer RL algorithms (DAPO, GSPO, CISPO) on vision‑language data.

Limitations / open items

• Vision‑language models are not yet compatible with the ulysses parallelism feature.
• No scripts for pure supervised fine‑tuning or inference; the authors recommend using LlamaFactory for those tasks.

Community & support

• Discussion via a WeChat group (link in README).
• Frequently asked‑question section for common CUDA/OOM and data‑size errors.
• A growing list of downstream projects (MMR1, Vision‑R1, Seg‑Zero, etc.) that cite EasyR1.

Citation

@misc{zheng2025easyr1,
  title        = {EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework},
  author       = {Yaowei Zheng and Junting Lu and Shenzhi Wang and Zhangchi Feng and Dongdong Kuang and Yuwen Xiong and Richong Zhang},
  howpublished = {\url{https://github.com/hiyouga/EasyR1}},
  year         = {2025}
}