DaoyuanLi2816/mini-verl

verl for a single consumer GPU. PPO, GRPO and on-policy distillation on NVIDIA GPUs.

What it solves

mini-verl allows users to run advanced reinforcement learning (RL) training workflows—specifically PPO, GRPO, and on-policy distillation (OPD)—on a single consumer-grade NVIDIA GPU. It bridges the gap for developers who want to use verl configurations and training recipes but lack the massive compute clusters typically required for these methods.

How it works

The project implements a sequential execution model where different roles (Actor, Critic, Reference, Reward, and Teacher) share a single GPU in phases rather than running in parallel across a cluster. It supports standard verl configuration trees and Hydra overrides, enabling users to train locally and then export adapters, datasets, and configurations for scaling out to larger setups.

Who it’s for

It is designed for developers and researchers who want to experiment with RL training locally on a single GPU, those who already use verl and want a local testing ground, and users looking to perform on-policy distillation or reward-based training on consumer hardware.

Highlights

  • Single-GPU Efficiency: Optimized to run complex RL workloads (like GRPO and PPO) on consumer hardware (e.g., RTX 4080).
  • verl Compatibility: Directly supports verl configuration profiles and overrides.
  • Exact Recovery: Ability to resume training from checkpoints, restoring the model, optimizer, data position, and random state.
  • Flexible Training Recipes: Supports PPO, GRPO, on-policy distillation (OPD), SFT, DPO, and offline KD.
  • Exportable Results: Generates trained adapters, reward logs, and readiness reports for easy hand-off to larger compute environments.

Related

  • Project
    verl-project/verl

    A flexible and efficient RL training library for LLMs that simplifies the implementation of RLHF algorithms and scales to trillion-parameter models.

  • Project
    leggedrobotics/rsl_rl

    A GPU-accelerated, lightweight learning library for robotics research that enables rapid prototyping of learning algorithms with minimal overhead.

  • Project
    Devsh-Graphics-Programming/Nabla

    A low-level GPU programming framework based on Vulkan and SPIR-V that provides a curated feature profile and a shared HLSL/C++ library for cross-platform graphics development.

  • Project
    OpenRLHF/OpenRLHF

    A high-performance RLHF framework that combines Ray and vLLM to scale reinforcement learning from human feedback for LLMs and VLMs.

  • Project
    ServiceNow/PipelineRL

    A scalable asynchronous reinforcement learning framework for LLM agents that uses in-flight weight updates to maximize GPU utilization while maintaining on-policy data freshness.