Benchmarking Text Generation Inference

Hugging Face has released a benchmarking tool for Text Generation Inference (TGI) that allows developers to profile the trade-offs between throughput and latency. This tool enables users to move beyond simple throughput metrics to determine the optimal configuration for specific use cases, such as Retrieval-Augmented Generation (RAG) or chat, thereby reducing deployment costs and improving user experience.

Understanding LLM Inference Performance

LLM inference is fundamentally inefficient because decoders require a new forward pass for every token generated. To mitigate this, TGI incorporates several performance-enhancing techniques, including Flash Attention, Paged Attention, quantization, and speculation. However, the optimal configuration for these tools depends heavily on the specific workload:

  • RAG Use-Cases: Typically involve longer prompts (due to multiple retrieved documents) and medium-sized outputs, often pushing the limits of the model's context window.
  • Chat Scenarios: Generally involve shorter prompts and outputs across multiple turns.

Key Performance Metrics

To effectively profile a deployment, the TGI benchmarking tool focuses on four primary metrics:

  • Token Latency: The time required to process and send a single token to the user.
  • Request Latency: The total time taken to fully respond to a request.
  • Time to First Token (TTFT): The duration from the initial request to the return of the first token, combining pre-fill processing and the generation of the first token.
  • Throughput: The number of tokens the server can return within a set amount of time.

Throughput and latency are orthogonal measurements; optimizing for one often requires a trade-off with the other.

The Two-Phase Inference Process

LLM generation occurs in two distinct stages, which the benchmarking tool profiles separately:

  1. Pre-filling Stage: The entire prompt is processed in a single forward pass to generate the first token. This stage is critical for determining the TTFT.
  2. Decoding Stage: Each subsequent token is generated one by one. The generated token is appended to the input for the next forward pass. This stage is where the majority of the computation time is typically spent.

Using the TGI Benchmarking Tool

The benchmarking tool is integrated into TGI and can be run via a CLI. Hugging Face provides a dedicated Space that combines a TGI Docker image with a Jupyter Lab environment to facilitate testing.

Analyzing Results

The tool generates scatter plots where the X-axis represents latency (lower is better) and the Y-axis represents throughput (higher is better). The ideal performance point is the top-left corner of the chart.

  • Vertical Data Trends: If the curve is vertical, throughput can be increased (by increasing batch size) without degrading latency. This indicates the system has headroom to handle more users "for free."
  • Horizontal Data Trends: If the curve is horizontal, the system is compute-bound. Increasing the batch size increases latency for all users without providing a corresponding gain in throughput, signaling a need for better configuration or hardware scaling.

Deployment Strategy

Profiling should be an iterative process. Developers are encouraged to estimate user behavior, run benchmarks, and then refine TGI settings based on the results. These findings can then be applied to equivalent hardware on AWS, GCP, or Hugging Face Inference Endpoints to ensure the deployment is cost-effective and performant.

Sources