vLLM Encoder Disaggregation (EPD) Scaling and Multimodal Serving

Overview of EPD Disaggregation

Multimodal inference requires an encoding stage to transform media (images, video, audio) into embeddings before language model execution. Because the demand for encoding can vary independently of prefill and decode requirements, vLLM supports Encoder/Prefill/Decode (EPD) disaggregation. This architecture separates these stages into independently deployed vLLM instances, allowing operators to optimize resource allocation based on specific workload needs.

Metadata-Only Requests

To improve efficiency, vLLM now supports metadata-only requests. In this workflow, the encoder produces structural metadata (such as image grids or feature lengths) alongside embeddings. Downstream consumers use this metadata to reconstruct the input layout without needing to re-process or re-fetch the raw media. This reduces redundant computation and network overhead. For audio workloads, metadata-only forwarding has been shown to increase throughput from 62.21 to 73.13 requests/s while reducing mean latency from 980 ms to 853 ms.

Dedicated Encoder Execution

Encoder-only instances in vLLM are now optimized through a dedicated MMEncoderModelRunner. Key improvements include:

  • Reduced Startup I/O: Encoder-only instances skip loading language-model checkpoint shards, drastically reducing initialization time.
  • CUDA Graph Support: The encoder-only runner supports opt-in ViT graph capture for improved execution speed.
  • Asynchronous Execution: Encoder steps are isolated from token sampling, preventing buffer contention and ensuring consistent performance.
  • GPU Preprocessing: Integration of GPU-based preprocessing and NVDEC hardware decoding has demonstrated significant throughput gains in video-heavy workloads.

Remote Embedding Transfer and Reuse

Efficiently moving embeddings between disaggregated components is critical for performance. vLLM supports two primary remote-transfer paths:

  • NIXL: Shares CPU-offloaded encoder outputs between instances via registered CPU mmap regions.
  • Mooncake: Delivers encoder outputs using the Mooncake TransferEngine, supporting both TCP and RDMA configurations.

Additionally, the Mooncake Store enables cross-encoder output reuse. If an image is encoded by one instance, the resulting embedding can be cached and reused by other encoders, eliminating redundant forward passes. Experiments with InternVL3_5-38B-HF showed that shared reuse reduced repeated image encodings to zero in reassignment scenarios, significantly improving Time To First Token (TTFT).

Orchestration and Deployment

vLLM provides several integration paths for EPD orchestration:

  • vLLM Router: Supports native CLI configuration for E+PD and E/P/D deployments, including multi-image distribution.
  • SMG: Offers gRPC-based EPD support with both Python and Rust servicers, capable of distributing image processing across multiple encoder workers.
  • llm-d: Provides a Kubernetes-native deployment path where encoder and consumer roles run as separate pod tiers, allowing for independent scaling and hardware selection.

Performance Impact

In evaluations using DeepSeek V4.1 Flash on GB200 hardware, EPD deployments demonstrated 6.6–9.5% higher throughput per GPU compared to a standard PD baseline at high concurrency (64–128). Furthermore, P90 TTFT was reduced by 44–47%, highlighting the benefits of separating the encoding stage for compute-intensive multimodal tasks.

Sources