Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1

Asynchronous RL with Kernel Alignment

Asynchronous reinforcement learning (RL) offers significant speed advantages by allowing rollout and learner processes to run concurrently, but it traditionally suffers from weight lag and kernel mismatch. The vLLM team, in collaboration with NVIDIA Research and Inferact, has introduced REINFORCE Zero, a method that stabilizes asynchronous RL by eliminating kernel mismatch. By ensuring that rollout and training compute identical logprobs for the same weights and tokens, the remaining weight lag can be corrected using simple, unclipped importance sampling (IS) without the need for complex rejection gates or truncation.

The Two Sources of Logprob Gaps

The gap between rollout and learner logprobs consists of two distinct components:

  1. Weight Lag: This occurs because the learner updates on samples generated by older model versions. This gap changes smoothly over time and is effectively managed by importance sampling.
  2. Kernel Mismatch: This arises when rollout and training compute different numerical results for the same weights due to differences in batch composition or execution. Unlike weight lag, this fluctuates significantly and persists despite IS, leading to training instability.

By aligning kernels, the kernel mismatch term is reduced to zero. This allows unclipped IS to maintain stable training even at high policy lags, such as lag 64, where samples may be generated by weights 64 versions older than the learner's current state.

REINFORCE Zero Methodology

REINFORCE Zero simplifies the RL training loop into three core steps:

  • Kernel Alignment: Ensuring the rollout engine and the learner produce bit-for-bit identical logprobs for identical inputs.
  • Unclipped Token-Level IS: Using the ratio of learner probability to rollout probability directly as a weight, without truncation or filtering.
  • Plain REINFORCE Optimization: Calculating advantages based on the mean reward of a group and averaging tokens within a response, avoiding the need for separate value models or rejection gates.

This approach has demonstrated stability over 20,000 updates and achieves 2.40× the training-step throughput of synchronous training with aligned kernels.

Support for Flagship Models

Zero-mismatch RL has been extended to support major flagship models, ensuring bit-for-bit consistency between rollout and training:

  • DeepSeek-V4.1-Flash: Achieved zero mismatch at release precision, including support for FP8, MXFP4, and sliding window attention. Rollout steps remain within 10% of the fast baseline.
  • Kimi K3.1: Achieved zero mismatch on day-0, including support for KDA linear attention and LoRA adapters.

Cross-Platform Implementation: AMD MI355X

Zero mismatch is architecture-agnostic. On AMD MI355X hardware, the team achieved bit-for-bit consistency by addressing batch-invariant issues in libraries like hipBLASLt and optimizing expert-parallel operations. For DeepSeek-V4-Flash, these optimizations reduced serving-throughput loss significantly, and for Kimi K3.1, they enabled efficient RL training with LoRA adapters on multiple nodes.

All recipes, fixes, and implementation details are available in the Inferact/zero-mismatch-rl repository, with the REINFORCE Zero paper and code hosted at hijkzzz/REINFORCE-Zero.

Sources