vLLM achieves near-zero-downtime scaling in Elastic Expert Parallelism with NIXL EP
vLLM has introduced Elastic Expert Parallelism (EP) with near-zero downtime by implementing NIXL EP and a two-phase reconfiguration flow, allowing large language models to scale GPU resources without interrupting serving performance.
The Challenge of Traditional Scaling in Expert Parallelism
Traditionally, scaling a vLLM service required adding or removing entire model replicas or restarting the engine with a different GPU count. This process resulted in significant serving downtime, which is unacceptable for production inference workloads.
For instance, restarting vLLM to scale DeepSeek V4 Pro from 16 to 20 GB200 GPUs resulted in a service outage lasting 197 seconds. Even previous efforts to avoid a full restart still incurred substantial pauses: scaling from 16 to 20 GPUs took 100 seconds, and scaling back to 16 GPUs took 56 seconds.
Elastic EP Reconfiguration Architecture
The new Elastic EP system reorganizes the reconfiguration flow into two distinct phases: background preparation and a short commit. This architecture ensures that existing workers continue serving requests while the necessary changes are being prepared in the background.
Background Preparation: Existing workers utilize separate background threads to prepare and warm up the target communication, transfer model weights, and synchronize execution details (like autotuning results) with the joining workers. Concurrently, joining workers initialize, receive model weights, and prepare their communication and GPU execution.
Short Commit Phase: During the commit phase, vLLM briefly pauses its schedulers, activates the prepared topology, and resumes serving. This pause is minimized by the technical innovations detailed below.
NIXL EP: Expanding Communication In Place
Prior EP communication backends in vLLM used fixed membership, meaning adding or removing ranks required creating an entirely new communication instance. This led to two major problems: increased communication memory usage due to maintaining old and new instances, and the invalidation of previously captured CUDA graphs.
NIXL EP was designed specifically for Elastic EP to solve these issues by adding or removing ranks in place, reusing the same NIXL agent, buffers, and connections. This allows new peers to connect while existing workers continue serving.
Crucially, NIXL EP keeps captured EP dispatch and combine entries valid across reconfiguration. This prevents the invalidation of CUDA graphs, which previously required repeating warmup and graph construction during the commit phase, resulting in pauses of 58 seconds during scale-up and 44 seconds during scale-down, even with asynchronous preparation.
Preserving CUDA Graphs on Existing Workers
CUDA graphs record GPU operations, allowing vLLM to replay a forward pass without repeatedly launching operations from the CPU. Preserving these graphs across scaling requires maintaining stability beyond just the communicator object.
To achieve this, vLLM implemented several changes to keep the captured arguments, launch configurations, and input and output tensor addresses stable during reconfiguration:
- Retained workers reuse their fused-MoE execution objects instead of rebuilding them for a different active EP size.
- The addresses and layouts of graph-visible workspaces and dispatch outputs remain stable, ensuring captured expert kernels use the correct pointers and offsets.
- The backing storage of expert-routing tensors is preserved, allowing Expert Parallel Load Balancing (EPLB) to change mapping contents without moving the tensors used by captured kernels.
These changes enable retained workers to reuse their existing CUDA graphs after both scale-up and scale-down.
Independent Preparation for Joining Workers
Joining workers typically require other workers to participate in model forward passes for preparation. NIXL EP allows each joining worker to temporarily mask every remote EP peer during warmup and graph preparation. This isolation enables the joining worker to run its dummy batches and capture CUDA graphs using only its local rank active for EP communication.
This modification allows joining workers to capture CUDA graphs without requiring existing workers to participate in the preparation process, which is critical while existing workers are serving real requests.
Performance Benchmarks on DeepSeek V4 Pro
Experiments using DeepSeek V4 Pro on GB200 GPUs (TP=1, DP/EP sizes of 16 and 20) demonstrated the dramatic reduction in serving downtime achieved by NIXL EP.
| Metric | A: Blocking | B: Async preparation | C: Async preparation + graph reuse |
|---|---|---|---|
| 16 to 20 GPUs: downtime | 99.62 s | 58.04 s | 0.82 s |
| 20 to 16 GPUs: downtime | 56.25 s | 44.07 s | 1.96 s |
| 16 to 20 GPUs: total time | 99.72 s | 97.27 s | 92.34 s |
| 20 to 16 GPUs: total time | 56.21 s | 51.46 s | 9.32 s |
Configuration C, which utilizes NIXL EP, achieved near-zero downtime, reducing the scale-up pause from 58.04 seconds to 0.82 seconds and the scale-down pause from 44.07 seconds to 1.96 seconds.
Furthermore, in an end-to-end serving scenario under rising request load, Elastic EP remained below the service-level objective (SLO) of a rolling p95 time to first token below 5 seconds throughout the scaling process. During preparation, Elastic EP matched the static 16-GPU deployment's serving performance, and after scaling, it matched the recovered 20-GPU deployment's performance.
Future Directions
The vLLM team has outlined several next steps to further enhance the Elastic EP capability:
- Restore Warm Workers with Dynamo Snapshot: Integrating Dynamo Snapshot will allow joining workers to restore initialized process and CUDA state, eliminating the need to repeat compilation, kernel warmup, and graph construction.
- State-of-the-Art EP Communication: The team aims to achieve state-of-the-art dispatch and combine performance with NIXL EP while maintaining full support for Elastic EP and fault tolerance.
- Orchestrator Integration: The next step involves connecting Elastic EP to orchestrator autoscalers, integrating vLLM with systems like Dynamo and llm-d.