Hugging Face Accelerated Inference API Optimization

Hugging Face has implemented a series of optimization strategies to achieve a 100x performance increase for transformer inference within its Accelerated Inference API. This improvement is critical for deploying large-scale NLP models into production environments where real-time consumer application latency is a primary constraint.

High-Level Library and Pipeline Optimizations

Hugging Face achieves an initial 10x speedup by utilizing the most efficient methods within the Transformers and Tokenizers libraries, independent of the target hardware.

Efficient Model Pipelines

The API utilizes optimized model pipelines to reduce computation during each forward pass. For GPT-based text-generation tasks, this involves reducing the dimensionality of attention matrices computation by focusing exclusively on the new attention of the last token in each pass.

Rust-Based Tokenization

Because tokenization is frequently a bottleneck for inference efficiency, Hugging Face employs the Rust implementation of model tokenizers via the 🤗 Tokenizers library. Combined with smart caching, this approach provides up to a 10x speedup in overall latency.

Hardware-Specific Compilation and Low-Level Tuning

To move beyond initial gains and achieve further 10x performance boosts, Hugging Face applies low-level optimizations that are specific to the model architecture and the target hardware (CPU or GPU).

Static Graph Optimizations for CPU

For CPU-based inference, the team employs static graph techniques to maximize compute efficiency. These include:

  • Graph Optimization: Removing unused data flow within the model graph.
  • Layer Fusion: Combining layers using specific CPU instructions.
  • Quantization: Optimizing operations to reduce precision and compute requirements.

Custom ONNX Runtime Implementations

While open-source tools like the ONNX Runtime can be used with Transformers, Hugging Face notes that out-of-box functions may lead to suboptimal results or significant accuracy loss, particularly during quantization. The team achieves additional speedups by manually tuning the Transformers code and ONNX Runtime configurations to align with specific model architectures.

Scaling for Large-Scale Model Architectures

The rapid increase in model size—growing from BERT's 110 million parameters to GPT-3's 175 billion parameters—has made production deployment increasingly challenging. Hugging Face leverages its position as the maintainer of the Transformers and Tokenizers libraries, alongside partnerships with hardware and cloud vendors including Intel, NVIDIA, Qualcomm, Amazon, and Microsoft, to tune the intersection of models and infrastructure using the latest hardware optimization techniques.

Sources