Hugging Face Assisted Generation for Low-Latency Text Generation

TL;DR

Hugging Face has introduced Assisted Generation, a new decoding method designed to reduce the latency of autoregressive text generation. By using a smaller, faster assistant model to propose candidate tokens that a larger model then validates in a single forward pass, latency can be reduced by up to 10x on commodity hardware, particularly when using memory offloading.

The Bottleneck of Text Generation Latency

Text generation latency is primarily a memory-bandwidth problem rather than a compute problem. In a standard autoregressive forward pass, the bottleneck occurs when loading model layer weights from GPU RAM to the GPU compute cores. Because large models require hundreds of sequential forward passes to generate a full response, this memory transfer overhead becomes the dominant cost.

While existing optimizations like Flash Attention, INT8 quantization, batching (to increase throughput), and Tensor Parallelism (to distribute memory bandwidth) exist, they often come with high monetary costs or trade-offs in latency. Assisted Generation addresses this by reducing the total number of forward passes required from the primary model.

How Assisted Generation Works

Assisted Generation leverages a specific property of language decoders: a single forward pass can be used to validate a sequence of tokens rather than just predicting the next one. If a sequence of tokens is passed to the model without caching, the model returns logits for every position in that sequence.

The Assisted Generation Loop

The process operates as a cycle between a small assistant model and a large primary model:

  1. Candidate Generation: A small assistant model uses greedy decoding to generate a short sequence of candidate tokens (starting with a 5-token window).
  2. Validation: The primary model performs a single forward pass on these candidates to obtain logits for each position.
  3. Comparison: The primary model's predicted tokens are compared to the assistant's candidates from left to right.
  4. Correction: The first mismatch identifies where the assistant failed. The primary model's prediction at that mismatch point is kept, and all subsequent assistant candidates are discarded.
  5. Adjustment: A heuristic adjusts the number of candidates requested for the next iteration: increasing the window by 2 if all tokens matched, and decreasing it by 1 if any mismatch occurred.

Requirements for the Assistant Model

To maintain efficiency, the assistant model must meet two criteria:

  • Shared Tokenizer: The assistant must use the exact same tokenizer as the primary model to avoid expensive CPU-based decoding and re-encoding steps.
  • Size Differential: The assistant should be at least an order of magnitude smaller than the primary model to ensure that its generation time is negligible compared to the primary model's forward pass.

Performance and Capabilities

Assisted Generation provides varying levels of speedup depending on the hardware and model configuration:

  • Memory Offloading: The most significant gains are seen when models do not fit in GPU memory and rely on offloading, with speedups up to 10x.
  • GPU-Resident Models: When the model fits in GPU memory, speedups range from 2x (standard) to 3x (with INT8 quantization).
  • Task Suitability: The method is most effective for input-grounded tasks such as automatic speech recognition (ASR), translation, and summarization.

Compatibility with Sampling

While designed for greedy decoding, Assisted Generation can be used with multinomial sampling. However, the effectiveness of the assistant depends on the temperature setting. Low temperatures (close to 0) approximate greedy decoding and retain most latency benefits, while high temperatures increase randomness, causing the assistant to fail more often and reducing speedups.

Future Directions and Related Work

Hugging Face suggests that the future of text generation may move away from the fixed compute cost per token. Instead, architectures could be designed where different parts of a sequence are generated by models of varying sizes based on the complexity of the text.

This approach shares core principles with other research, specifically Blockwise Parallel Decoding (Google Brain) and Speculative Sampling (DeepMind), both of which utilize forward passes to validate longer continuations.

Sources