vLLM Elastic Expert Parallelism achieves near-zero-downtime scaling
Traditional scaling of large language models required restarting the service or adding/removing entire model replicas, resulting in significant serving downtime. vLLM's new Elastic Expert Parallelism (Elastic EP) architecture, powered by NIXL EP, allows the addition or removal of GPUs in a running deployment with minimal disruption, reducing scaling downtime to under 2 seconds.
Scaling Architecture: Asynchronous Preparation and Commit
The core of the near-zero-downtime capability is the reorganization of vLLM's reconfiguration flow into two distinct phases: background preparation and a short commit. During the background preparation phase, existing workers continue serving requests while separate threads prepare the target topology, transfer model weights, and warm up new workers. The commit phase is a brief pause where vLLM activates the prepared topology before resuming serving.
Preparation involves four main tasks executed asynchronously:
- Initialize joining workers: Starting new worker processes and allocating necessary model and execution storage.
- Prepare communication: Setting up the target communication groups and expanding the Expert Parallel (EP) communicator.
- Transfer model weights: Moving required model weights from existing workers to joining workers via NVLink or RDMA.
- Prepare execution: Completing kernel compilation, warmup, tuning setup, and CUDA graph construction on the joining workers.
After the commit phase, Asynchronous Expert Parallel Load Balancing (EPLB) takes over, allowing new workers to handle attention while existing workers manage expert computation, followed by background redistribution of experts.
NIXL EP Enables In-Place Communication Expansion
NIXL EP is a specialized EP communication library designed to solve the limitations of fixed-membership communication backends. Previously, adding or removing ranks required creating an entirely new communication instance, which led to increased memory usage and invalidated existing CUDA graphs.
NIXL EP solves these issues by adding or removing ranks in place, reusing the same NIXL agent, buffers, and connections. This allows new peers to connect while existing workers continue serving. Crucially, NIXL EP keeps captured EP dispatch and combine entries valid across reconfiguration, meaning retained workers can reuse their existing CUDA graphs without needing to repeat warmup and graph construction.
During preparation, vLLM's background thread connects joining peers without activating them. The commit phase then activates the prepared peers without replacing the core communicator.
Preserving CUDA Graphs on Existing Workers
CUDA graphs record GPU operations, allowing vLLM to replay a forward pass without repeated CPU-side launches. Preserving these graphs during scaling requires maintaining stability across the reconfiguration process. NIXL EP ensures that kernel launch configurations, graph-visible communication arguments, and input/output tensor addresses remain stable across scaling events.
Specific changes implemented in vLLM to preserve these graphs include:
- Reusing retained workers' fused-MoE execution objects instead of rebuilding them for a different active EP size.
- Keeping the addresses and layouts of graph-visible workspaces and dispatch outputs stable, ensuring captured expert kernels use correct pointers even if the EP size changes.
- Preserving the backing storage of expert-routing tensors and updating mappings in place, allowing EPLB to change mappings without moving the tensors used by captured kernels.
These measures enable retained workers to reuse their existing CUDA graphs after both scale-up and scale-down.
Independent Preparation for Joining Workers
Joining workers must prepare their execution and capture CUDA graphs, but they cannot rely on existing workers, which are actively serving real requests. NIXL EP facilitates independent preparation by allowing each joining worker to temporarily mask every remote EP peer during warmup and graph preparation. This isolates the worker's local rank for EP communication.
Additionally, vLLM was modified so that these preparation passes do not wait for Data Parallel (DP) batch information from other workers. This allows joining workers to run dummy batches and capture CUDA graphs without requiring existing workers to participate in the preparation process.
Performance Benchmarks on DeepSeek V4 Pro
Experiments were conducted using DeepSeek V4 Pro on GB200 GPUs, comparing three configurations: Blocking (A), Async preparation (B), and Async preparation + graph reuse (C). Configuration C, supported only by NIXL EP, demonstrated significant performance improvements.
Downtime Reduction
| Metric | A: Blocking | B: Async preparation | C: Async preparation + graph reuse |
|---|---|---|---|
| 16 to 20 GPUs: downtime | 99.62 s | 58.04 s | 0.82 s |
| 20 to 16 GPUs: downtime | 56.25 s | 44.07 s | 1.96 s |
Scaling from 16 to 20 GPUs reduced the observed serving interruption (downtime) from 99.62 seconds to 0.82 seconds. Scaling back from 20 to 16 GPUs reduced downtime from 56.25 seconds to 1.96 seconds.
Service Level Objective (SLO) Maintenance
In an end-to-end serving scenario with a rising request load, Elastic EP maintained the service-level objective (SLO) of a rolling p95 time to first token below 5 seconds throughout the scaling process. During preparation, Elastic EP matched the serving performance of the static 16-GPU deployment, and after scaling, it matched the performance of the recovered 20-GPU deployment.
Future Directions
Future development focuses on two primary areas: accelerating worker startup and improving communication performance.
- Restore Warm Workers with Dynamo Snapshot: The next challenge is reducing the time required for new capacity to become ready. The team plans to integrate Dynamo Snapshot so joining workers can restore initialized process and CUDA state instead of repeating compilation, kernel warmup, and graph construction.
- State-of-the-Art EP Communication: The goal is to achieve state-of-the-art dispatch and combine performance with NIXL EP while maintaining full support for Elastic EP and fault tolerance.