Faster Text Generation with TensorFlow and XLA

TL;DR

Hugging Face has enabled XLA (Accelerated Linear Algebra) compilation for text generation within the transformers library for TensorFlow. This optimization can increase generation speed by up to 100x and, in many benchmark cases, outperforms PyTorch for text generation tasks.

Accelerating TensorFlow with XLA

XLA is a compiler designed to accelerate TensorFlow models and is also the foundation for JAX and certain PyTorch implementations. In TensorFlow 2, which uses Eager Execution for better transparency and debugging, some performance advantages of graph mode are lost. To recover these advantages, users can wrap functions in tf.function, which converts the code into a graph.

By adding the jit_compile=True argument to tf.function or tf.keras.Model.compile, users can trigger XLA compilation. While the first call to an XLA-compiled function is slow due to the compilation process, subsequent calls with the same tensor shapes and types are significantly faster.

Implementation Requirements for XLA Text Generation

XLA relies on just-in-time (JIT) compilation and polymorphism. To avoid costly re-compilation (tracing) during text generation, the following technical requirements must be met:

Input Padding

Because XLA triggers a new compilation step whenever it encounters a different tensor shape, type, or non-tensor argument, input prompts must be padded to a consistent length. Hugging Face recommends using the pad_to_multiple_of argument in tokenizer classes to balance input flexibility with a limited number of possible shapes.

Codebase Vectorization

Auto-regressive text generation is inherently dynamic, often expanding tensors and using dynamic slices, which are not XLA-friendly. To enable XLA support, Hugging Face rewrote the TensorFlow text generation codebase to vectorize operations and utilize fixed-sized structures with padding. Additionally, NLP models were modified to ensure positional embeddings function correctly with these padded structures.

Text Generation Capabilities in Transformers

The generate function in the transformers library supports several decoding strategies:

  • Greedy Decoding: The default deterministic approach (do_sample=False) that picks the most likely token at each step.
  • Sampling: A stochastic approach (do_sample=True) where randomness can be controlled via the temperature setting. Lower values prioritize high-likelihood tokens, while higher values increase entropy.
  • Beam Search: Triggered when num_beams is greater than 1, this method explores high-probability sequences to improve output quality over greedy decoding.

Performance Benchmarks

Benchmarks comparing TensorFlow and PyTorch across multiple GPU models show two primary results:

  1. Massive Speedups: TensorFlow text generation is significantly faster when XLA is used, with speedups exceeding 100x in some instances.
  2. Framework Comparison: In the vast majority of cases, TensorFlow with XLA is the fastest option, sometimes performing up to 9x faster than PyTorch for text generation tasks.

Sources