Accelerate BERT Inference with Hugging Face Transformers and AWS Inferentia

TL;DR

Hugging Face has demonstrated how to accelerate BERT inference by compiling Transformers models using the AWS Neuron SDK and deploying them on AWS Inferentia chips via Amazon SageMaker. This approach reduces latency to 5-6ms for sequence lengths of 128 and offers higher throughput and lower costs compared to traditional GPU-based EC2 instances.

AWS Inferentia for Transformer Acceleration

AWS Inferentia is a custom machine learning chip designed specifically for optimized inference workloads. According to AWS, Inferentia can deliver up to 80% lower cost per inference and up to 2.3X higher throughput than comparable current-generation GPU-based Amazon EC2 instances.

The architecture utilizes "Neuron Cores," which are custom accelerators within the chip. Each Inferentia chip contains four Neuron Cores, allowing users to either:

  • Load one model per core to maximize throughput.
  • Load one model across all cores to minimize latency.

Technical Workflow for BERT Deployment

Deploying a BERT-like model (such as distilbert-base-uncased-finetuned-sst-2-english) on AWS Inferentia involves a multi-step compilation and deployment process.

1. Model Conversion via AWS Neuron SDK

The AWS Neuron SDK provides a deep learning compiler and runtime to convert PyTorch and TensorFlow models into neuron-compatible formats for EC2 Inf1 instances.

Because the AWS Neuron SDK does not support dynamic shapes, the input size must be static during compilation and inference. For example, if a model is compiled with a batch size of 1 and a sequence length of 128, it can only process inputs of that exact shape.

2. Custom Inference Scripting

While the Hugging Face Inference Toolkit supports zero-code deployments for many models, AWS Inferentia currently requires a custom inference.py script. This script must define:

  • model_fn: To load the tokenizer, the neuron model, and the model configuration.
  • predict_fn: To handle input embeddings, ensure they match the static sequence length (via padding and truncation), and execute the prediction.

To maximize throughput, the environment variable NEURON_RT_NUM_CORES=1 is used to ensure each HTTP worker utilizes a single Neuron core.

3. SageMaker Deployment

Once the model is compiled and the inference script is created, the artifacts are archived into a model.tar.gz file and uploaded to Amazon S3. The model is then deployed as a real-time inference endpoint using the HuggingFaceModel class in Amazon SageMaker, specifically targeting ml.inf1.xlarge instances.

Performance Results

In a load test consisting of 10,000 synchronous requests, the BERT model achieved an average latency of 5-6ms for a sequence length of 128.

Hugging Face concludes that this setup is faster than CPU-based inference and provides higher throughput than GPUs by running four models in parallel across the Neuron Cores. This makes AWS Inferentia a viable option for companies using BERT-like Transformers for encoder tasks such as text classification, token classification, and question answering.

Sources