OpenRLHF/OpenRLHF

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

What it solves

OpenRLHF provides a high-performance, production-ready framework for Reinforcement Learning from Human Feedback (RLHF). It addresses the scalability and efficiency bottlenecks of RLHF training—specifically the fact that sample generation typically consumes 80% of the training time—by integrating a distributed architecture that optimizes both inference and training.

How it works

The framework uses a distributed architecture combining Ray for scheduling and resource management, vLLM for high-throughput sample generation, and DeepSpeed for memory-efficient training (ZeRO-3).

It introduces a unified agent-based design paradigm where every training run is treated as a "token-in-token-out" agent execution. This decouples the RL algorithms from the execution mode, allowing users to switch between:

  • Single-Turn Mode: Standard one-shot generation for most RLHF use cases.
  • Multi-Turn Mode: Multi-step interactions with environment feedback, suitable for complex reasoning or interactive agents.

Who it’s for

It is designed for AI researchers and engineers who need to align large language models (LLMs) and Vision-Language Models (VLMs) using RLHF at scale, ranging from small models to those with hundreds of billions of parameters.

Highlights

  • Diverse RL Algorithms: Supports PPO, REINFORCE++, GRPO, RLOO, and FlashREINFORCE.
  • Hybrid Engine Scheduling: Allows models and vLLM engines to share GPU resources to maximize utilization.
  • Broad Model Support: Compatible with HuggingFace Transformers, including support for LoRA/QLoRA, MoE, and VLMs.
  • Advanced Optimizations: Includes sample packing, RingAttention for long contexts, and the Muon optimizer.
  • Flexible Reward Shaping: Features DAPO-style overlong penalties and ProRL-style truncation penalties for better length control.

Related

  • Project
    verl-project/verl

    A flexible and efficient RL training library for LLMs that simplifies the implementation of RLHF algorithms and scales to trillion-parameter models.

  • Project
    lucidrains/PaLM-rlhf-pytorch

    A PyTorch implementation of RLHF on the PaLM architecture, providing tools to train a base model, a reward model, and align them using a reinforcement learning trainer.

  • Dispatch
    Illustrating Reinforcement Learning from Human Feedback (RLHF)

    Hugging Face explains Reinforcement Learning from Human Feedback (RLHF), a three-step process used to align large language models with complex human values by optimizing them using a reward model based on human preferences.

  • Project
    whitecircle/halo

    A high-performance training framework for LLMs and multimodal models that integrates with Hugging Face and supports advanced parallelism and asynchronous RL.