vLLM Native RL APIs Release

vLLM has released native Reinforcement Learning (RL) APIs designed to standardize weight syncing between training and inference and stabilize asynchronous RL workloads. These updates resolve common issues where weight syncing was implemented in ad-hoc, duplicated ways across frameworks and where asynchronous RL setups became fragile at scale, particularly in P/D and DPEP deployments.

Native Weight Syncing APIs

To ensure rollouts in online RL are generated from the most recent model weights, vLLM now provides a standardized interface for weight transfer, replacing the need for framework-specific worker extensions.

Weight Transfer Lifecycle

The weight transfer process is divided into four distinct phases with a pluggable backend:

1. Initialization (init_weight_transfer_engine): Establishes the communication channel between the trainer and inference workers before the training loop begins.
2. Start weight update (start_weight_update): Prepares vLLM workers to receive weights, typically called after each training step.
3. Update weights (update_weights): Transfers all or a subset of weights from the trainer to the inference engine, supporting chunked transfers.
4. Finish weight update (finish_weight_update): Executes necessary post-processing, such as quantization.

Supported Backends and Customization

vLLM currently supports two primary backends for weight transfer, both utilizing optimized packed implementations to reduce serialization overhead:

• NCCL: Used for weight transfer between training and inference workers on separate GPUs via broadcast operations.
• IPC: Uses CUDA IPC for same-device weight transfer through shared memory handles.

Developers can implement their own transport logic by registering a custom WeightTransferEngine abstraction, separating the transport mechanism from the worker implementation.

Improved Asynchronous RL Support

Asynchronous RL requires updating weights while inference requests are still in flight. vLLM has introduced new mechanisms to handle this process without interrupting client requests or causing system deadlocks.

Introduction of "Keep Mode"

vLLM has added a keep mode to the pause_generation and resume_generation methods (and corresponding /pause and /resume HTTP APIs). This allows the engine to pause ongoing requests and stop the scheduler while preserving the state, ensuring clients do not need to retry requests.

Mode	Explanation	Client-side impact	Asynchronous RL possible?
abort	Aborts all ongoing requests	Client must handle retries	Yes
wait	Wait for all ongoing requests	Client need not retry	No
keep	Pause ongoing requests	Client need not retry	Yes

Resolving Deadlocks in DPEP Setups

In Data Parallelism (DP) deployments, deadlocks previously occurred when some engines received a pause signal while others were actively handling requests and waiting for synchronization. vLLM addresses this through two primary changes:

1. EngineCore Integration: Pause logic has been moved from the AsyncLLM entrypoint layer directly into the scheduler within EngineCore to reduce race conditions.
2. Two-Phase Pause/Resume Protocol:
   • Phase 1 (Local Pause): Engines pause scheduling but continue to respond to START_DP_WAVE requests to participate in required forward passes.
   • Phase 2 (Global Pause): Engines perform a periodic all-reduce (every 32 steps) to verify if all ranks are in the "local pause" state. Once consensus is reached, they transition to a global pause together.

Validation and Performance

Framework Integration

The new APIs have been integrated into SkyRL, enabling asynchronous training for Qwen3-1.7B using the DAPO recipe.

Large-Scale Deployment

The Prime-RL team validated the APIs using a zai-org/GLM-5.1-FP8 model in a P/D disaggregated setup. The deployment spanned 16 8xH200 nodes (2 replicas of 4P+4D with DPEP32 for prefill and decode) and utilized CPU KV cache offloading (1TB per node). The setup remained stable over 100+ training steps with consistent weight updates and improving evaluation performance.