vLLM Prefill-Decode Disaggregation with MORI-IO

vLLM has introduced Prefill-Decode (PD) disaggregation for single-node deployments, enabling a 2.5x increase in goodput on an 8-GPU AMD Instinct MI300X node. By separating the compute-bound prefill phase from the memory-bandwidth-bound decode phase, vLLM eliminates the Inter-Token Latency (ITL) spikes that typically occur when these workloads compete for the same GPU resources.

The Bottleneck of Collocated Serving

In standard monolithic deployments, prefill and decode workloads share the same GPU resources and scheduler. Because prefill processes entire prompts in parallel using large GEMMs, it is compute-bound and takes significantly longer than a single decode step. When a prefill request enters a batch, it blocks all ongoing decode streams, causing unpredictable spikes in Inter-Token Latency (ITL).

Single-Node PD Disaggregation Architecture

Contrary to the common assumption that disaggregation requires multi-node clusters, vLLM implements this architecture within a single 8-GPU node. The system shifts to a microservice architecture consisting of three components:

• Prefill Instance: Processes input prompts and produces the KV cache (e.g., using GPUs 0–3).
• Decode Instance: Generates output tokens using the transferred KV cache (e.g., using GPUs 4–7).
• Proxy Server: Acts as the entry point, routing requests first to the prefill instance and then to the decode instance.

To facilitate the transfer of gigabytes of KV cache data between these instances, vLLM utilizes MORI-IO, an RDMA-based KV cache connector built on the open-source MORI (Modular RDMA Interface) framework.

KV Cache Transfer Modes: Read vs. Write

MORI-IO supports two distinct transfer modes, configured via the VLLM_MORIIO_CONNECTOR_READ_MODE environment variable:

Read Mode (VLLM_MORIIO_CONNECTOR_READ_MODE=1)

In Read Mode, the proxy dispatches requests serially. It waits for the prefill instance to complete and return remote_block_ids before forwarding the request to the decode instance. The decode instance then pulls the KV cache from the prefill instance's memory via RDMA.

Write Mode (Default)

In Write Mode, the proxy dispatches requests to both the prefill and decode instances concurrently. As the prefill instance computes each layer, it pushes the KV data directly into the decode instance's pre-allocated memory via RDMA WRITE. This eliminates the proxy serialization overhead and allows the decode queue wait to overlap with prefill computation.

Property	Read Mode	Write Mode
RDMA Direction	Decode pulls from prefill	Prefill pushes to decode
Proxy Dispatch	Serial (await prefill $\rightarrow$ dispatch decode)	Concurrent (prefill and decode in parallel)
Block ID Relay	Required via proxy	Not required
KV Cleanup	Decode notifies prefill to free blocks	Prefill tracks write completion

Performance Results and Goodput

Using the Qwen3-235B-A22B-FP8 model on an 8-GPU MI300X node (2000-token prompts, 1000-token outputs), vLLM measured performance using goodput—the maximum request rate where requests satisfy both Time to First Token (TTFT) < 1s and ITL < 50ms.

Key Findings

• 2.5x Goodput Increase: Write mode achieved 73/100 requests meeting SLOs at a rate of 8 req/s, compared to only 26/100 for standard TP8 serving.
• Elimination of ITL Spikes: Disaggregated modes entirely eliminated ITL violations, as the decode engine is isolated from compute-intensive prefill jobs.
• TTFT Trade-off: Disaggregation increases TTFT due to the added RDMA transfer and proxy overhead. Write mode provides better TTFT than Read mode because it eliminates proxy serialization.

Deployment and Configuration

Deploying PD disaggregation requires configuring the kv-transfer-config in vLLM. Both instances must specify the MoRIIOConnector and their respective roles (kv_producer for prefill and kv_consumer for decode).

Port Requirements

• proxy_ping_port: ZMQ endpoint for instance registration with the proxy.
• http_port: vLLM HTTP server port for inference requests.
• handshake_port: One-time metadata exchange for KV cache layout.
• notify_port: Per-request synchronization signaling when KV blocks are ready.

Summary of Use Cases

Situation	Recommendation
ITL p99 exceeds SLO under production load	Disaggregate
TTFT is the primary binding constraint	Standard serving
High concurrency with long prompts	Disaggregate
Low request rates with short prompts	Standard serving