Optimizing LLM Performance: Solving Long Prompt Blocking and Decode Slowdowns

TL;DR

Long prompts in LLM serving can block the prefill queue and slow down token generation for concurrent requests. To mitigate this, request-parallel prefills reduce time-to-first-token for short prompts, while disaggregated prefill isolates prefill and decode phases on separate GPUs to eliminate interference and stabilize latency.

The Challenge of Long Prompt Blocking

In standard LLM serving, the prefill phase (processing the initial prompt) is compute-intensive and can saturate GPU utilization, whereas the decode phase (generating subsequent tokens) is less compute-intensive. In the default chunked-prefill strategy used by vLLM, prefill chunks for different requests are scheduled sequentially.

When a request with a very long prompt is scheduled, it blocks the prefill queue. Any subsequent requests must wait for the long prefill to complete before their own prefill phase can begin, significantly increasing the time-to-first-token (TTFT) for those requests.

Request-Parallel Prefills

To address queue blocking, vLLM has implemented a strategy allowing parallel prefills for multiple requests, subject to a limit on the number of concurrently processed long prompts (e.g., allowing four parallel prefills but only one exceeding 10,000 tokens).

  • Impact on Short Prompts: Short prompts can now bypass long prefills via a "fast lane," drastically reducing their TTFT.
  • Impact on Long Prompts: Long prompts continue to be processed sequentially to avoid severe system slowdowns that would occur if multiple compute-intensive prefills were batched together.
  • Limitation: While TTFT is reduced, the time-per-output-token remains elevated because the concurrent prefill still consumes GPU resources, slowing down the decode steps of existing requests.

The Fundamental Flaw: Prefill-Decode Interference

Executing prefill and decode operations for different requests within the same GPU operation causes a slowdown in token generation. A single request with a long prompt is sufficient to degrade the performance of all previously scheduled requests currently in their decode phases.

Mitigation Strategies

There are three primary ways to handle this interference:

  1. Priority Penalization: Long prompts can be forced to wait until high-priority or short requests are finished. This increases latency for long prompts and does not solve the interference issue once a long prompt is actually scheduled.
  2. Dedicated Inference Servers: Routing long-prompt requests to a separate server. This requires a sophisticated router and additional GPU resources, though short-context servers can be deployed on fewer GPUs (e.g., Llama-3.3-70B requires four H100s for 130k context, but only two H100s for <10k context).
  3. Disaggregated Prefill: Using separate inference engines for prefill and decode. This architecture involves multiple vLLM deployments where one worker handles only prefill and another handles only decode. Once prefill is complete, the KV cache is transferred to the decode worker.

Disaggregated Prefill for Latency Optimization

Disaggregated prefill eliminates the direct disruption of decode phases caused by concurrent prefills, making it the most effective strategy for stabilizing token generation latency.

Trade-offs and Current State

  • Resource Cost: This approach requires separate full-size vLLM deployments for each role (e.g., eight H100s for Llama-3.3-70B: four for prefill and four for decode).
  • GPU Utilization: Utilization is often uneven because prefill is more compute-intensive than decode. However, large clusters can balance this by adjusting the ratio of prefill to decode workers based on load patterns.
  • Goal: The primary objective is to increase "goodput" (the rate of requests meeting latency targets) rather than total raw throughput.
  • Experimental Status: As of vLLM v0.7.3, the feature is experimental. Current limitations include lower context length limits and inconsistent use of CUDA graphs in the decode worker, which can result in slower decode speeds compared to integrated deployments.

Sources