vLLM Push-Based NIXL KV Connector: Up to 3x Better TTFT for Disaggregated Inference

vLLM has added a push-based NIXL KV connector for prefill-decode (P-D) disaggregated inference that moves transfer coordination and setup off the critical path to first token. In a Qwen3-32B benchmark on a 1P1D deployment using Amazon EC2 P5en instances connected with AWS EFA, push mode delivered up to 3x better time-to-first-token (TTFT) and up to 30% better time-per-output-token (TPOT) compared with the default pull mode.

What problem the push connector solves

Prefill-decode disaggregation runs prefill and decode on separate engine instances so each phase can scale and be optimized independently, improving resource utilization, throughput, and latency under load. The cost is that the KV cache must be transferred from the prefill instance to the decode instance before token generation can begin, which puts KV transfer directly on the TTFT critical path.

vLLM performs this transfer through the NVIDIA Inference Xfer Library (NIXL) connector, which moves KV blocks over RDMA-capable interconnects such as AWS Elastic Fabric Adapter (EFA). On supported instance types, EFA supports GPUDirect RDMA and Neuron Direct RDMA, so NIXL can transfer KV blocks directly between GPU or Trainium device memory without staging through host memory.

The default NIXL connector mode is pull-based. After the prefill instance finishes computing the prompt KV cache, the router relays source metadata, including prefill block IDs, to the decode instance. The decode instance then allocates destination blocks, prepares the NIXL transfer, and initiates an NIXL READ to pull the KV blocks from the prefill instance. Because these coordination and setup steps run sequentially after prefill and before decoding can begin, they extend the TTFT critical path.

How push-based KV transfer works

Push-based transfer improves TTFT and decode throughput for two reasons stated in the vLLM post:

1. It removes the serialization of the pull model by overlapping producer and consumer work. The decode instance registers its target blocks in advance while the producer is still computing KV. The transfer begins as soon as the producer's blocks are ready, rather than after a consumer-side identifier-discovery round trip, removing that round trip from the path to first token.
2. It moves the transfer-setup time off the decode instance. Preparing the NIXL descriptors and submitting the transfer runs on the prefill side and on a background thread, so the decode instance does not spend those cycles during its latency-critical token-generation phase.

The net effect is lower TTFT and higher output-token throughput on the decode instance.

The push connector coordinates three actors: the decode instance, which registers its target blocks up front; the prefill instance, which matches each request and issues the transfer; and a dedicated background thread that carries the KV write off the engine's forward-pass loop.

The three-step protocol

1. Registration (decode side). When the decode instance allocates local blocks for a request requiring remote prefilling, it prepares a registration describing where the prefill instance should write the KV: its own identity and the block locations it has just reserved. Within the same engine step, the worker transmits this registration to the prefill instance as a NIXL notification. Because registration happens at allocation time, the connector knows the destination blocks before the prefill instance completes prefill.

2. Rendezvous (prefill side). The registration from the decoder and the completion of prefill on the producer can arrive in either order, and the connector handles both. If the registration arrives first, the producer holds it until its own prefill finishes. If prefill finishes first, the producer holds the request's blocks until the registration arrives. Whichever event happens second triggers the transfer. The connector pairs requests by identifier, and the matching accounts for reissued requests, correctly associating preempted or retried requests.

3. Transfer (prefill side). Once a request is matched, the producer establishes a connection to the decode instance (if one does not already exist) and issues an NIXL WRITE that copies the KV blocks directly into the decoder's registered memory, addressing each destination tensor-parallel rank. The producer and consumer may use different internal block layouts; each side maps block locations to its own physical layout at transfer time, allowing different block sizes on each side. Each WRITE carries a completion notification that NIXL delivers to the decode instance once the data arrives. The decoder counts these notifications, one per producer rank that writes into it, and treats the KV as received only after all expected notifications have arrived, at which point the request proceeds to decode.

Threading model

The push connector separates model execution from KV-transfer preparation by using a dedicated background transfer thread on each prefill worker. At the start of a request, the decode main loop allocates its destination KV blocks and registers them with the prefill instance while the prefill main loop computes the prompt KV cache. After both the destination registration and prompt KV are ready, the prefill main loop hands the KV blocks to the background thread and quickly returns to model execution.

The background thread establishes or reuses the NIXL connection, maps the request's KV block IDs to memory descriptors, merges contiguous blocks into larger transfer regions, and submits the NIXL WRITE to the decode instance. When NIXL completes the transfer, it sends a completion notification to the decode instance, which then begins token generation. Keeping descriptor preparation and transfer submission off the latency-critical main loops prevents this per-request work from delaying model execution.

Pull vs push: the routing tradeoff

Both modes use the same NIXL and EFA/RDMA data path. Pull performs the metadata relay and decode-side setup after prefill, while push overlaps decode-side setup with prefill, removing those operations from the post-prefill critical path so decoding can begin sooner.

The two modes also involve a routing tradeoff. Pull mode allows the router to select a decode instance after prefill completes, which provides greater flexibility for load-aware routing and quality-of-service management. Push mode selects the decode instance earlier so that destination setup can overlap with prefill. In return, under comparable load, the prefill instance can release its KV blocks sooner because the transfer starts as soon as both the prompt KV and destination registration are ready.

Performance data

The evaluation used the Qwen3-32B model on a 1P1D deployment using Amazon EC2 P5en instances connected with EFA. To isolate P→D transfer performance, the benchmark disabled prefix caching with --no-enable-prefix-caching, forcing the decode node to retrieve the full KV cache from the prefill node for every request. Without that setting, overlapping prompts could reuse the decode node's local cache and bypass the transfer, producing misleading comparison results.

TTFT. The benchmark swept input lengths of 1k–16k and request rates of 1–8 QPS. At low load and short prompts, the two modes perform almost identically. As QPS and input length grow, pull mode degrades sharply, reaching about 1,744 ms at 8 QPS and 16k input. Push mode stays far lower at roughly 540 ms, around a 3x reduction. The gap opens because pull mode posts NIXL descriptors inline on the decode worker, synchronously within its engine loop, so that work stalls token generation; push mode moves it to a background thread, off the critical path.

TPOT. Under the same conditions, the pattern mirrors TTFT. At light load, push and pull are close, around 5.6–5.8 ms. As load and input length rise, pull climbs to about 10.4 ms at 4–8 QPS and 16k input, while push holds near 7 ms. Because push offloads descriptor preparation and transfer submission from the decode worker, token generation is never starved, so per-token latency stays flat even at high concurrency and long contexts.

How to enable push mode

To use push mode, configure both the prefill and decode servers to use NixlPushConnector instead of NixlConnector. The vLLM post states that the feature was built and validated on AWS Trainium instance clusters connected with EFA and then contributed to vLLM as an accelerator-agnostic feature; the published results come from an AWS p5en (GPU) cluster with EFA, showing the gains carry over to GPUs.

When to choose push vs pull

Choose the push-based connector when early decode node selection is acceptable and lower TTFT is the priority. The pull-based connector remains the better fit for workloads that require bidirectional KV transfer or that defer decode-instance selection until after prefill for load-aware routing. vLLM supports both push- and pull-based KV connectors, allowing users to select the approach that best fits their deployment.

Reproducing the benchmark

The reference topology is two p5en GPU nodes (prefill on :8100, decode on :8200) with EFA enabled, plus a third smaller node running the proxy and the benchmark client. Both GPU nodes used p5en.48xlarge (8x NVIDIA H200), Ubuntu 24.04.4 LTS with kernel 6.14.0-1018-aws, NVIDIA driver 580.126.16 (CUDA 13.0 driver API), CUDA toolkit 13.0, and EFA installer 1.50.0 (EFA kernel module 3.3.0g, libfabric 2.6.0amzn1.0, rdma-core 64.0amzn0). Each instance exposes 16 EFA interfaces at 200 Gb/s each, for 3.2 Tb/s total.

Software versions in the reproduction guide are vllm==0.29.0 and nixl==1.4.1. The mode difference is a single config field: NixlConnector for pull, NixlPushConnector for push. The decode server adds --no-enable-prefix-caching to force full KV fetches from prefill. The benchmark script lives in vLLM pull request #57567, and the post ships serve_pd.sh, run_sweep.sh, cleanup.sh, and plot_push_vs_pull.py for launching, sweeping, tearing down, and redrawing the figures.

On EFA device selection, the LIBFABRIC backend opens all 16 EFA devices as rails with the efa provider, then registers each GPU's KV cache only on the 2 rails that share that GPU's PCIe switch, so every transfer stays PCIe-local. No NIXL or libfabric device overrides were set; this is the default selection.

Future work

vLLM is exploring bringing the background NIXL threading model to the pull-mode connector, moving NIXL setup off the critical path so pull-mode workloads could benefit from the same optimizations. A pull request (#45211) has been opened to explore this.