vLLM RL-Kernel v0.1.0: Bitwise Train-Rollout Consistency on ROCm

vLLM and RL-Kernel have implemented a system to ensure bitwise numerical consistency between training and rollout logprobs on AMD Instinct MI300X GPUs. This integration eliminates the numerical policy shift that typically occurs when different execution engines handle training and inference, ensuring that the importance ratio and KL diagnostics in RL post-training are grounded in identical floating-point computations.

The Problem of Train-Rollout Numerical Mismatch

In reinforcement learning (RL) workflows, training and rollout typically utilize different execution engines and operators. Even when using the same model weights and inputs, differences in kernel implementations, parallelism strategies, and reduction orders can lead to divergent logprobs.

This mismatch introduces a numerical policy shift that propagates into the importance ratio, rollout-training KL diagnostics, and clipping mechanisms. While previous efforts to solve this have focused on NVIDIA platforms, this work extends those guarantees to the ROCm ecosystem.

Technical Architecture: vime and RL-Kernel

Achieving consistency requires aligning both the timing of data flow and the actual numerical execution of the kernels. The workload is divided between two components:

  • vime: Aligns the training timeline. It manages which token batch belongs to which step, tracks which weight version generated specific tokens, and synchronizes new weights to vLLM.
  • RL-Kernel: Aligns the numerical execution. It ensures that the same values participate in computations and that they are partitioned, merged, and rounded identically across both engines.

The Numerical Execution Contract

To achieve bitwise consistency, RL-Kernel enforces a strict numerical execution contract across the forward path. This contract defines six key observable behaviors:

  1. Input Dependency: The specific elements and reduction domain the output depends on.
  2. Partitioning: How the reduction domain is divided into partial results.
  3. Merge Order: The sequence in which partial results are combined.
  4. Precision: The precision of inputs, accumulators, intermediate states, and outputs.
  5. Rounding: The exact points where rounding or downcasting occurs.
  6. Primitives: The specific numerical primitives used (e.g., exp, log, rsqrt, SiLU, and FMA).

Implementation on ROCm

Adapting these guarantees to AMD Instinct MI300X required modifying several native ROCm execution paths to ensure they matched the training side's behavior. Key technical fixes included:

  • GEMM Kernels: Implementation of deterministic MFMA-based GEMM kernels.
  • Attention: Fixing the execution schedule of AITER/CK Attention and addressing paged KV layout state issues.
  • Communication: Implementing HIP IPC communication and fixing distributed collective rank reduction orders.
  • Math and Fusion: Resolving last-bit differences caused by compiler fusion and specific math function implementations.
  • Logprobs: Implementing vocabulary reduction and fixing cross-TP vocabulary merges.

Boundary Fixes in the RL-Kernel Strict Path

Compute Boundary Strict Path Fixes
RMSNorm Reduction domain, epsilon, residual addition, and output boundary
Attention Position, mask, logical paged KV, split policy, LSE precision, and final cast
GEMM and SwiGLU K-dimension reduction, accumulation precision, epilogue, activation, and materialization boundary
Linear logprob True vocabulary range, target ownership, local reduction, and cross-rank LSE merge
Distributed collectives Payload ownership, dtype, and a fixed rank reduction order

Validation Results on AMD Instinct MI300X

An end-to-end experiment was conducted using a Qwen3-8B model (BF16) on a single node with 8x AMD Instinct MI300X GPUs. The setup utilized Megatron (TP4/CP2/PP1) for training and two vLLM engines (TP4 each) for rollout.

Key Findings

  • Zero Mismatch: Over 200 consecutive rollout and training steps, the mismatch_count and max_abs_diff remained exactly zero.
  • Comparison to Native vime: While native vime showed mismatches at every step, the vime + RL-Kernel configuration maintained bitwise identity throughout the trajectory.
  • Stability: The mean absolute train-rollout logprob difference remained at zero for the duration of the 200-step run.

Summary of Achievements

The integration of vime and RL-Kernel on ROCm provides several critical capabilities for RL post-training:

  • Bitwise Consistency: Exact matching of training and rollout logprobs on ROCm hardware.
  • Verifiable Reproducibility: Stable consistency guarantees that make RL training trajectories easier to verify.
  • Failure Localization: The ability to use operator ablations to identify the exact system boundary where numerical divergence begins.
  • Execution Evidence: Comprehensive validation of kernels, HIP Graph execution, and paged KV paths used at runtime.

Sources