vLLM Disaggregated Serving Guide

vLLM has introduced disaggregated serving to separate the three primary workloads of LLM inference: prompt processing (prefill), token generation (decode), and CPU-bound frontend tasks (tokenization and parsing). By splitting these stages, vLLM prevents long prompts from stalling active generation streams, allowing operators to tune time to first token (TTFT) and inter-token latency (ITL) independently.

Technical Architecture of Disaggregation

Disaggregated serving in vLLM splits the inference pipeline into distinct tiers that can be scaled and tuned independently:

Prefill and Decode Separation

Prefill and decode run as separate instances. The prefill instance processes the initial prompt and produces the KV cache (attention keys and values), which is then transferred to the decode instance via a KV connector.

KV Cache Transfer: The size of the transfer depends on the model and prompt length. For example, Llama-3.1-70B in BF16 stores 320 KiB per token; a 10k-token prompt requires transferring approximately 3 GB of data.
Connectors: vLLM supports multiple connectors for this transfer, including NIXL, LMCache, Mooncake, FlexKV, and AMD's MoRI-IO, often utilizing RDMA for high-speed movement.

Frontend Disaggregation (Render/Derender)

CPU-intensive tasks are moved to a GPU-less frontend tier:

/render: Converts OpenAI-style requests into token IDs.
/derender: Converts output token IDs back into a structured OpenAI response, including the separation of content, reasoning, and tool calls.

Other Disaggregation Dimensions

Encoder Disaggregation: Used for multimodal models to separate the encoder from the rest of the pipeline.
AFD Plugin: Splits attention from Feed-Forward Networks (FFN) in Mixture-of-Experts (MoE) models.
Hybrid SSMs: Supports Mamba state transfer for hybrid SSM models.

Performance Impact: Goodput and Latency

Disaggregated serving focuses on improving goodput—the request rate sustainable while meeting both TTFT and ITL targets—rather than peak raw throughput.

Reducing Tail Latency

In collocated serving, a long prefill request can stall all concurrent decode streams on the same GPU, causing spikes in p99 ITL. In a disaggregated setup, decode instances only run decode batches, eliminating prefill-induced stalls.

In tests using Qwen2.5-7B-Instruct on two NVIDIA L40S GPUs with 8k-token prompts, collocated p99 ITL jumped from 23 ms to 169 ms at 0.4 req/s, whereas the disaggregated (P/D) setup's p99 ITL remained between 25 ms and 52 ms.

Increasing Goodput

Fast KV transfer is critical for these gains. AMD's MoRI-IO benchmark with Qwen3-235B-A22B-FP8 showed that 73% of requests met TTFT (<1s) and ITL (<50ms) targets in disaggregated mode, compared to only 30% in collocated mode—a 2.4× increase in goodput.

The Cost of Transfer

Every first token pays a latency penalty for the KV cache transfer. If the transfer is slow (e.g., lacking peer-to-peer support or RDMA), TTFT will increase. On hardware without P2P support, 8k-token prompts took ~1.3s to transfer, causing the system to miss TTFT targets even while ITL remained stable.

Operational Implementation

Running Prefill/Decode

Starting in vLLM v0.30.0, a disaggregated setup requires three components: a prefiller, a decoder, and a proxy to orchestrate the handoff.

Prefiller: Configured as a kv_producer using a specific kv_connector (e.g., NIXL).
Decoder: Configured as a kv_consumer.
Proxy: Calls the prefiller with max_tokens=1 and do_remote_decode: true, then forwards the resulting kv_transfer_params to the decoder.

Multi-Turn Conversations

To avoid recomputing the KV cache for previous turns in a chat, vLLM supports bidirectional_kv_xfer. This allows the prefiller to pull existing blocks back from the decoder, computing only the new tokens in the prompt.

Deploying the Render Tier

The render tier is stateless and scales as a standard CPU web service. It is critical to keep the model, tokenizer, and parser configurations synchronized across the render, derender, and engine tiers to avoid output mismatches.

Current Limitations and Future Work

Streaming Parser Overhead: Streaming derender with a parser is computationally expensive because it currently rebuilds the parser and replays token history for every chunk, resulting in $O(n^3)$ character work over long generations. A caching layer is planned to address this.
Wire Overhead: Long prompts dominate the request body because prompt_token_ids are resent with every chunk during streaming derender.
Correctness Gaps: Issues are being addressed regarding finish_reason: "tool_calls" in batch derender and the return of placeholder strings instead of integer token IDs for logprobs.

Summary Recommendation Table

Situation	Recommendation
ITL p99 misses SLO under load	Disaggregate
Long prompts at high concurrency	Disaggregate (if KV transfer is fast)
Chat/agent loops over growing context	Disaggregate with bidirectional transfer
GPU CPU profile shows high tokenization/parsing load	Split off the render tier
TTFT is the binding constraint	Stay collocated or measure first
Slow KV transfer	Fix transfer or stay collocated
Low, bursty, or latency-insensitive traffic	Stay collocated