Optimizing LLM Performance: Prefill and Decode for Concurrent Requests

Executive Summary

Optimizing Large Language Model (LLM) performance requires managing the fundamental difference between the prefill phase (processing the input prompt) and the decode phase (generating subsequent tokens). By implementing continuous batching and chunked prefill, developers can maximize GPU resource utilization and significantly increase total token throughput—TNG observed a 50% increase in throughput using chunked prefill in standard vLLM deployments.

The Two Stages of Token Generation

LLMs generate text auto-regressively, meaning each new token depends on all preceding tokens. This process is split into two distinct computational phases:

Prefill Phase

  • Function: Computes the first output token by processing all input prompt tokens.
  • Computational Profile: Highly parallelizable and GPU compute-intensive. Because all input tokens are known at the start, the model can calculate key and value vectors for the entire prompt simultaneously.
  • Key Metric: Measured as Time to First Token (TTFT).

Decode Phase

  • Function: Computes every subsequent token one by one.
  • Computational Profile: Sequential and memory-bandwidth bound. Only one set of key and value vectors needs to be calculated per token, but the model must access weights and the Key-Value (KV) cache from GPU memory for every step.
  • Key Metric: Measured as Time Per Output Token (TPOT).

Resource Utilization and Throughput

GPU utilization varies significantly between these two phases. A single request with a long prompt can saturate GPU compute power during the prefill phase. In contrast, the decode phase for a single request utilizes very little computational power, meaning throughput is increased by batching multiple requests together to fill the GPU's compute capacity.

Throughput typically increases linearly with concurrency at low levels (the memory-bound regime) until GPU compute power saturates (the compute-bound regime), at which point throughput remains invariant regardless of further increases in concurrency.

Concurrent Processing Strategies

Inference engines use different batching strategies to handle multiple requests arriving simultaneously, each with different trade-offs regarding latency and efficiency.

Static Batching

Static batching groups requests into a fixed batch, processes them until the longest request in the batch is complete, and only then starts a new batch.

  • Advantage: Optimizes TPOT because the decode phase is uninterrupted.
  • Disadvantage: Highly inefficient resource utilization and potentially very high TTFT, as new requests must wait for the entire previous batch to finish.

Continuous Batching (Prefill-First)

Continuous batching removes completed requests immediately and inserts new ones. A "prefill-first" strategy schedules new prefills as soon as they arrive.

  • Advantage: Minimizes TTFT by processing new requests immediately.
  • Disadvantage: Interrupts the decode phase of existing requests. Because a prefill operation dominates GPU execution time, concurrent requests in the decode phase may only generate a single token during the entire duration of a large prefill, causing "pauses" in streamed output.

Chunked Prefill

Chunked prefill breaks the input prompt into smaller chunks distributed over multiple steps. This allows multiple decode steps to occur during the prefill process.

  • Advantage: Maximizes resource efficiency by running compute-intensive prefills and memory-bound decodes in parallel. This reduces the average TPOT by preventing complete pauses in token generation.
  • Impact: TNG reported a +50% increase in total token throughput in standard vLLM deployments using chunked prefill.
  • Trade-off: Introduces a small increase in TTFT and requires tuning the chunk size (typically between 512 and 8192 tokens) to balance the priority between TTFT and TPOT.

Sources