DaoyuanLi2816/mini-verl

What it solves

miniVERL allows users to run complex Reinforcement Learning from Human Feedback (RLHF) experiments—specifically those designed for the verl framework—on a single NVIDIA GPU. It removes the need for massive distributed clusters by "lowering" the experiment semantics onto one card, making high-end RL training accessible for local development and testing.

How it works

The project acts as a temporal scheduler. Instead of running the actor, critic, reference model, and reward model simultaneously across multiple GPUs, miniVERL schedules these logical roles to run in phases on a single GPU. It includes a versioned compiler that takes upstream verl configuration trees and translates them into a validated single-GPU execution plan, preserving the original experiment's logic while managing memory and compute resources.

Who it’s for

It is designed for researchers and developers using the verl framework who want to prototype, debug, or run small-scale RL experiments locally without needing a distributed infrastructure.

Highlights

• Broad Algorithm Support: Supports PPO/GAE, GRPO, Dr.GRPO, RLOO, and REINFORCE++.
• Config Compatibility: Directly accepts upstream verl config trees and Hydra overrides without requiring a new configuration language.
• Portable Artifacts: Exports PEFT adapters, safetensors, and Parquet files that can be moved back into larger, distributed workflows.
• Exact Recovery: Uses transactional manifests and checkpoints to restore the policy, optimizer, and RNG state for byte-identical reproduction.
• Memory Efficient: Capable of running models like Qwen3-0.6B on consumer hardware (e.g., RTX 4080) with low VRAM overhead.