RL-Align/RL-Kernel

High-performance RL post-training infrastructure. Designed to achieve bitwise operator-level train-inference consistency across heterogeneous engines and extreme memory efficiency for GRPO, PPO, etc.

What it solves

RL-Kernel addresses the numerical inconsistency that occurs during Reinforcement Learning (RL) post-training. In standard setups, the rollout engine (which generates samples) and the training engine (which updates the model) often use different kernels or calculation orders. This leads to discrepancies in log probabilities, which can degrade the stability and quality of RL training. RL-Kernel ensures these two stages produce identical results, maintaining numerical consistency throughout the training process.

How it works

It acts as a middleware layer between execution engines (like vLLM for rollout and Megatron-LM for training) and the underlying hardware. By providing a set of deterministic, high-performance operators—such as attention, dense FFN, and specific RL objectives like GRPO and PPO—it forces both the rollout and training engines to perform calculations in a mathematically identical way. It also includes hardware-specific kernels to optimize performance and reduce memory usage on supported accelerators.

Who it’s for

This project is designed for machine learning engineers and researchers working on large-scale RL post-training for LLMs who need high-performance, reproducible, and cross-hardware training pipelines.

Highlights

  • Deterministic Consistency: Guarantees bitwise-accurate log probability agreement between rollout and training engines.
  • Cross-Hardware Support: Native support for NVIDIA (CUDA) and AMD (ROCm) GPUs, with partial support for Huawei Ascend hardware.
  • Performance Optimized: Uses fused computation and custom kernels to minimize rollout time and synchronization overhead.
  • Engine Integration: Integrates with vime to orchestrate vLLM and Megatron-LM, providing a unified path for RL workflows.

Related

  • Dispatch
    vLLM RL-Kernel v0.1.0: Bitwise Train-Rollout Consistency on ROCm

    vLLM and RL-Kernel have achieved bitwise numerical consistency between training and rollout logprobs on AMD Instinct MI300X GPUs, eliminating numerical policy shifts in RL post-training.

  • Project
    verl-project/verl

    verl is a flexible, high-performance reinforcement learning library designed for training and fine-tuning large language models at scale by decoupling computation and data dependencies.

  • Project
    verl-project/verl-omni

    VeRL‑Omni is an open‑source RL training framework for multimodal generative models (diffusion, unified, and omni‑modality). Built on verl and vLLM‑Omni, it provides fast rollout, asynchronous multi‑reward serving, distributed back‑ends, and many RL algorithms (FlowGRPO, DiffusionOPD, DPO, etc.). Supports GPUs and Ascend NPUs, with extensive docs, examples, and Apache‑2.0 licensing.

  • Project
    NVIDIA-NeMo/RL

    NeMo RL is an open-source post-training library for scalable reinforcement learning of large multimodal models (LLMs, VLMs). It provides algorithms like GRPO, DAPO, and SFT, with flexible backends for distributed training and generation.

  • Project
    PrimeIntellect-ai/prime-rl

    prime-rl is a framework for large‑scale reinforcement learning, designed for asynchronous RL training at scale. It enables training models on up to 1000+ GPUs with features like FSDP2, vLLM, and support for agentic and multimodal tasks.