Accelerating PyTorch Transformers with Intel Sapphire Rapids

Intel Sapphire Rapids CPUs, when paired with the Hugging Face Optimum Intel library, can accelerate PyTorch transformer inference by up to 3x compared to the previous Intel Xeon (Ice Lake) generation. This performance gain allows for single-digit prediction latency even with long text sequences, a benchmark previously primarily achievable via GPUs.

Strategic Advantages of CPU-based Inference

CPU-based inference is a viable and often superior alternative to GPU-based inference depending on the model size, task parallelism, and cost constraints.

  • Model Size: Smaller models typically run efficiently on CPUs, whereas larger models benefit more from the massive computational power of GPUs.
  • Parallelism: GPUs excel at massively parallel processing. If an inference task lacks high parallelism, a CPU may be more effective.
  • Cost and Flexibility: CPUs are often more cost-effective than GPUs, especially for use cases that do not require extremely low latency. They also provide greater flexibility for scaling inference workers and deploying across diverse hardware environments.

Hardware and Software Configuration

To evaluate the performance gains, benchmarks were conducted using Amazon EC2 instances with the following specifications:

  • Ice Lake (Baseline): c6i.16xlarge instance.
  • Sapphire Rapids: r7iz.16xlarge-metal instance.

Both instances utilized 32 physical cores (64 vCPUs) and were configured with Ubuntu 22.04, Linux 5.15.0, PyTorch 1.13, Intel Extension for PyTorch 1.13, and Transformers 4.25.1. The Sapphire Rapids instance additionally utilized the Optimum Intel library.

Benchmarking NLP Model Performance

Benchmarks were performed on text classification tasks using distilbert-base-uncased, bert-base-uncased, and roberta-base models. The tests measured mean and p99 prediction latency for both single and batch inference across short (16-token) and long (128-token) sequences.

Optimization Techniques

On the Sapphire Rapids architecture, the following optimizations were applied via the Optimum Intel library to leverage hardware capabilities:

  • bfloat16 Mode: Enabled to utilize Advanced Matrix Extensions (AMX) instructions.
  • Just-In-Time (JIT) Compilation: Set to True to further optimize model execution.

Performance Results

For the distilbert-base-uncased model, single predictions on Sapphire Rapids showed a 60-65% reduction in latency (running 60-65% faster) compared to the previous generation of Xeon CPUs. This combination of hardware and software allows for the achievement of single-digit prediction latency even when processing long text sequences.

Conclusion

The fourth generation of Intel Xeon CPUs provides a significant leap in inference performance for PyTorch Transformers. By integrating the Intel Extension for PyTorch and Hugging Face Optimum, developers can deploy deep learning models more cost-effectively and with performance levels that rival GPU-based solutions for specific NLP tasks.

Sources