OpenRLHF/OpenRLHF

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

What it solves

OpenRLHF is a high-performance, production-ready framework designed to make Reinforcement Learning from Human Feedback (RLHF) scalable and extensible. It addresses the bottleneck of sample generation—which typically consumes 80% of RLHF training time—by integrating a high-throughput inference engine with a distributed training architecture.

How it works

The framework utilizes a distributed architecture combining Ray for scheduling and model separation (Actor, Reward, Reference, and Critic models), vLLM for high-performance sample generation, and DeepSpeed ZeRO-3 for memory-efficient training.

Its core innovation is a unified agent-based design paradigm. This treats every training run as a "token-in-token-out" agent execution, allowing the same pipeline to support both single-turn RLHF (standard one-shot generation) and multi-turn RLHF (complex interactions with environment feedback). This design decouples the RL algorithms from the execution mode, meaning any supported algorithm can be used with any agent mode.

Who it’s for

It is built for researchers and engineers who need to train large-scale language models (up to 70B+ parameters) using RLHF, specifically those working on reasoning models, Vision-Language Models (VLMs), or interactive agents.

Highlights

  • Diverse RL Algorithms: Supports PPO, REINFORCE++, GRPO, and RLOO.
  • Agent-Based Execution: Unified pipeline for both single-turn and multi-turn interactions, including support for external environment feedback.
  • VLM Support: Capability to train Vision-Language Models with image inputs and multi-turn image feedback.
  • Hybrid Engine Scheduling: Maximizes GPU utilization by allowing models and vLLM engines to share resources.
  • Scalability: Integration with DeepSpeed AutoTP and RingAttention for long-context training and multi-node scaling via SLURM.

Related

  • Project
    opendilab/LightRFT

    A high-performance reinforcement learning fine-tuning framework for LLMs and VLMs that supports RLHF and RLVR with efficient inference engines and multimodal capabilities.

  • Project
    agentscope-ai/Trinity-RFT

    A general-purpose framework for reinforcement fine-tuning of LLMs that decouples exploration, training, and data buffering to support diverse RL algorithms and agentic workflows.

  • Project
    rllm-org/rllm

    An open-source framework for training language agents with reinforcement learning, allowing the same agent code to be used across different backends, sandboxes, and benchmarks.

  • Project
    langfengQ/verl-agent

    An extension of veRL for training LLM agents via reinforcement learning, featuring a step-independent rollout mechanism to efficiently handle long-horizon, multi-turn tasks.

  • Project
    inworld-ai/tts

    A training and modeling framework for SpeechLM-based text-to-speech systems, supporting pre-training, SFT, and RLHF alignment across single or multi-GPU clusters.