vLLM Bi-Directional KV Transfer Improves TTFT for Multi-Turn Agentic Workloads

TL;DR

vLLM adds bi‑directional KV cache transfer, allowing a decode node to hand its KV blocks back to the prefill node for subsequent turns, which reduces time‑to‑first‑token (TTFT) by up to 1.6× for Qwen3‑32B and 3.3× for GLM‑5‑FP8 in multi‑turn agentic workloads.

Why bi‑directional KV transfer matters

Multi‑turn agentic applications resend the entire conversation history on every turn, forcing the prefill node to recompute KV caches for previously generated tokens. This redundant work inflates accelerator usage and TTFT as the context grows. By enabling KV reuse across turns, vLLM eliminates most of that recomputation, keeping latency flat even for very long prompts.

Core technical contribution

Bi‑directional KV transfer mechanism

  • Two‑way KV flow – In addition to the traditional prefill‑to‑decode (P→D) NIXL READ, the decode node now retains eligible KV blocks after generation and returns their metadata to the router. The router attaches this metadata to the next prefill request, allowing a decode‑to‑prefill (D→P) NIXL READ.
  • Configurable controls –
    • decoder_kv_blocks_ttl (default 480 s) determines how long the decode node keeps KV blocks.
    • kv_recompute_threshold (default 64 tokens) decides whether to pull remote KV or recompute locally based on the reusable token count.

NIXL KV connector enhancements

  • Extended the NVIDIA Inference Xfer Library (NIXL) KV connector so prefill nodes can consume remote KV blocks, mirroring the existing decode‑side capability.
  • The actual data movement remains a single NIXL READ; only the control flow differs between P→D and D→P directions.

KV‑cache‑aware router

  • A lightweight, stateful proxy maintains an in‑memory map from conversation_id to the latest kv_transfer_params received from the decode node.
  • The router stores only metadata (block locations, TTL, expiry), never the KV tensors themselves, keeping the control plane lightweight.
  • Example implementation is provided in the vLLM repository (disagg_proxy_multiturn.py).

Performance impact

Benchmark setup

  • Hardware – 1 prefill + 1 decode node on Amazon EC2 P5en (8× NVIDIA H200) connected via AWS Elastic Fabric Adapter (EFA).
  • Models – Qwen3‑32B and zai‑org/GLM‑5‑FP8 (FP8).
  • Configuration – Prefix caching disabled on the prefill node to emulate cache‑eviction scenarios; kv_recompute_threshold = 64, decoder_kv_blocks_ttl = 480 s.

TTFT results

Model Prompt length (tokens) TTFT OFF TTFT ON Speed‑up
Qwen3‑32B 24 k 778 ms 460 ms 1.6×
GLM‑5‑FP8 20 k 2 334 ms 702 ms 3.3×
  • TTFT grows steeply when KV reuse is disabled (full recompute each turn).
  • With bi‑directional transfer, TTFT remains near‑flat; the modest KV‑transfer latency (≈30‑130 ms) is far outweighed by the saved recompute time.

Prefill duration breakdown

  • KV‑transfer time – 30‑130 ms for Qwen3‑32B; 29‑110 ms for GLM‑5‑FP8.
  • Prefill compute saved – The majority of the prefill leg disappears, especially for GLM‑5‑FP8 where per‑token recompute cost is higher.
  • Charts in the original post illustrate the stacked‑bar comparison for each prompt length.

How to adopt the feature

  1. Upgrade to vLLM v0.21.0+ (or later) and install the NIXL connector.
  2. Enable bidirectional_kv_xfer in the KV‑transfer config for both prefill and decode servers.
  3. Deploy a KV‑cache‑aware router – use the provided disagg_proxy_multiturn.py example or implement equivalent logic.
  4. Tune parameters – adjust decoder_kv_blocks_ttl to match expected inter‑turn latency and kv_recompute_threshold based on model size and network bandwidth.
  5. Run the benchmark – follow the reproducibility guide (scripts serve_pd_multiturn.sh, run_multiturn_sweep.sh, cleanup.sh) to validate performance on your own cluster.

Implications for agentic AI deployments

  • Scalable latency – Applications such as autonomous assistants, tool‑using agents, and multi‑step reasoning pipelines can maintain low latency even as conversation histories become thousands of tokens long.
  • Accelerator efficiency – By avoiding repeated KV recomputation, GPU/TPU cycles are freed for other workloads, reducing cost per request.
  • Hardware‑agnostic benefit – Although the feature was first validated on AWS Trainium clusters, the same gains appear on GPU‑based P5en instances, indicating broad applicability across accelerator types.
  • Future extensions – A lease‑renewal mechanism for decode‑side KV blocks could further reduce memory waste, and dynamic thresholds could adapt to varying network conditions.

Reproducibility checklist

  • Instances – Two p5en.48xlarge (8 × H200) with EFA enabled, same AZ.
  • Software stack – Ubuntu 24.04, NVIDIA driver 580+, CUDA 13.0, EFA 1.50.0, libfabric 2.6.0, NIXL 1.4.1, vLLM 0.29.0 (with PR #40186 applied).
  • Key commands – Installation of EFA, vLLM/NIXL, model download, server launch scripts, proxy start, benchmark sweep scripts.
  • Troubleshooting tips – Ensure conversation_id is sent, PR #40186 is applied, TTL is sufficient, and EFA provider is correctly detected.

Conclusion

Bi‑directional KV cache transfer lets a decode node hand back its KV blocks to the prefill node for the next turn, dramatically cutting TTFT for long, multi‑turn agentic conversations. In a 1 prefill + 1 decode deployment on AWS P5en GPUs with EFA, the feature yields 1.6× faster TTFT for Qwen3‑32B and 3.3× faster TTFT for GLM‑5‑FP8 at the longest prompts tested. The approach is accelerator‑agnostic, configurable via TTL and recompute thresholds, and can be integrated using vLLM’s NIXL connector and a KV‑cache‑aware router.

References:

Sources