Accelerating Hugging Face Transformers with AWS Inferentia2

TL;DR

Hugging Face has partnered with AWS to optimize Transformer models for AWS Inferentia2, a new purpose-built inference accelerator. This collaboration enables developers to deploy large-scale models with significantly lower latency and higher throughput without requiring complex manual model slicing or distribution techniques.

AWS Inferentia2 Capabilities and Performance

AWS Inferentia2 is the successor to Inferentia1, designed specifically to optimize the cost and speed of model inference. It provides substantial performance gains over both its predecessor and comparable GPU instances.

Performance Improvements

  • Comparison to Inferentia1: The Inferentia2 chip delivers a 4x increase in throughput and a 10x reduction in latency.
  • Comparison to NVIDIA A10G (G5 instances): Amazon EC2 Inf2 instances offer up to 2.6x better throughput, 8.1x lower latency, and 50% better performance per watt than comparable G5 instances.

Scalability and Memory

Inf2 instances are available in various sizes, containing between 1 and 12 Inferentia2 chips. These chips utilize direct chip-to-chip connectivity for distributed inference. The largest instance size, inf2.48xlarge, provides sufficient memory to load models with up to 175 billion parameters, such as GPT-3 or BLOOM.

Integration with Hugging Face

To reduce development complexity, Hugging Face provides the optimum neuron library. This integration, powered by the AWS Neuron SDK, allows users to compile models for Inferentia2 with a single line of code, removing the need for manual model modification or slicing.

Benchmarking Results

Hugging Face conducted 144 experiments across six model architectures (BERT-base, BERT-Large, RoBERTa-base, DistilBERT, ALBERT-base, and ViT-base) using batch size 1 and sequence lengths ranging from 8 to 512. The benchmarks compared inf1.2xlarge (Inferentia1), inf2.xlarge (Inferentia2), and g5.2xlarge (NVIDIA A10G GPU).

Key Latency Findings

On average, AWS Inferentia2 delivers 4.5x better latency than NVIDIA A10G GPUs and 4x better latency than Inferentia1 instances.

  • BERT-base: For sequence lengths up to 256, Inferentia2 outperforms other setups by approximately 6x.
  • Vision Transformer (ViT-base): Inferentia2 provides 2x better latency than the NVIDIA A10G, facilitating the transition from CNNs to Transformers for real-time applications.

Benchmark Configuration

Models evaluated on GPU were run in fp32 without additional optimization. The primary metrics measured were p95 latency (the time for a single prediction including pre- and post-processing) and throughput (the number of executions in a fixed time).

Sources