vLLM Train-Inference Consistency Release Notes

Overview

High-performance Reinforcement Learning (RL) for LLMs often suffers from train-inference mismatch, where numerical differences between the inference engine and the training engine cause the model to optimize a policy different from the one used for sampling. This mismatch can destabilize training, leading to reward collapse and increased reruns. vLLM has introduced a new approach that achieves bitwise-identical logprobs (diff = 0) between rollout and training, ensuring true on-policy RL while maintaining 91.3% of baseline rollout throughput on DeepSeek-V4-Flash.

The Technical Challenge: Train-Inference Mismatch

Rollout is the most time-consuming part of RL training, leading developers to adopt low-precision compute, low-precision KV caches, and various kernel optimizations. These optimizations often introduce non-deterministic behavior or numerical differences compared to the training forward pass.

In experiments with DeepSeek-V4-Flash, the baseline's train-inference KL divergence rose from 2e-3 to 29.9 over 100 steps, resulting in reward degradation. The vLLM approach solves this by forcing the training forward pass to use the exact same vLLM inference kernels as the rollout side, ensuring that both engines compute the same policy for the same weights.

Achieving Bitwise Consistency

To reach a diff of 0, the system enforces consistency across four dimensions:

  • Kernel Determinism: Ensuring the same kernel on the same input always produces the same output, avoiding non-associative floating-point accumulation issues.
  • Batch Independence: Removing dependencies where a request's result changes based on its batch-mates.
  • Phase Consistency: Unifying the computation path between prefill and decode phases.
  • Cross-GPU Consistency: Fixing communication reductions so they do not vary based on timing or scheduling.

By modifying high-performance kernels to meet these requirements, the team achieved bitwise-identical results without sacrificing the performance gains of low-precision operations.

Agent-Driven Implementation

To lower the cost of applying this consistency to new models, vLLM released an agent skill that automates the debugging and alignment process. The skill guides an agent through:

  1. Localization: Identifying the first operator where the two engines diverge.
  2. Minimal Repro: Reducing the divergence to a minimal, rerunnable test case.
  3. Verification: Ensuring the fix maintains consistency without breaking the high-performance baseline.

Results and Performance

  • DeepSeek-V4-Flash: The consistent version reached 0 diff at 91.3% of the baseline's rollout throughput. While the baseline reward fell back, the consistent version's reward remained stable.
  • Nemotron-3.5 (Mamba hybrid): The consistent version maintained a SWE-bench Pro score of 0.6, while the baseline score dropped from 0.56 to 0.22, demonstrating that consistency prevents training collapse in complex architectures.

Open Source Availability

The process, recipes, and code are now available to the community. This includes the train-inference-consistency skill for automated alignment, the DeepSeek-V4 recipe, and the Nemotron-3.5-Lightning-30B-A3B implementation. Development is currently ongoing for DeepSeek-V4.1, which has already achieved 0 diff with performance optimizations currently being finalized.

Sources