Scaling up BERT-like model Inference on modern CPU - Part 2

TL;DR

Hugging Face has demonstrated that optimizing software components—specifically memory allocators, parallelization libraries, and the use of Bayesian optimization—can significantly enhance the inference performance of BERT-like models on Intel Ice Lake Xeon CPUs. These optimizations, combined with hardware features like AVX-512 and VNNI, allow for more efficient scaling of NLP workloads on CPU-based infrastructure.

Leveraging Intel Software for AI Efficiency

To maximize the performance of AI workloads on Intel hardware, developers can leverage a stack of software optimizations under the Intel oneAPI umbrella. These tools are designed to bridge the gap between high-level frameworks like PyTorch and TensorFlow and the underlying CPU architecture.

Key Software Components

  • Intel oneMKL (Math Kernel Library): Provides highly efficient linear algebra routines.
  • Intel OpenMP and oneTBB (Threading Building Blocks): Frameworks for high-level parallelization of computations.
  • Intel oneDNN: A library of deep neural network primitives (e.g., ReLU, fully-connected layers) that is natively integrated into PyTorch and TensorFlow (since version 2.5.0).
  • Intel PyTorch Extension (IPEX): A specialized framework acting as a laboratory for optimizations before they are upstreamed to the main PyTorch library.

Performance Tuning Knobs for CPU Inference

Inference performance is primarily driven by three factors: data representation in memory, the implementation of mathematical operators, and the efficiency of parallelization.

Memory Allocation and Management

Memory allocation (the process of requesting dynamic memory from the OS) can impact speed and fragmentation. While default allocators like glibc are general-purpose, specialized allocators can reduce synchronization overhead in multi-threaded deep learning workloads:

  • tcmalloc (Google): Often provides the best performance across various workloads by maintaining local memory segments for each thread, reducing global critical paths.
  • jemalloc (Facebook): Can be the fastest in specific low-concurrency situations.
  • mimalloc (Microsoft): Another alternative for improving memory management.

Parallelization of Computations

Efficiently utilizing multiple CPU cores requires more than just increasing the core count. Factors such as CPU cache invalidation and concurrent data access can hinder scaling. Hugging Face recommends using the Intel implementation of the OpenMP specification ("IOMP") for Intel hardware to optimize thread dispatching and resource binding.

Optimized Mathematical Operators

Modern CPUs use SIMD (Single Instruction Multiple Data) instructions to operate on multiple items per clock cycle. Intel CPUs support SSE2, AVX, AVX2, and AVX-512. Libraries like Intel MKL and oneDNN implement these operators efficiently, enabling performance speedups for common patterns like Linear + ReLU or Convolution.

Benchmarking on Intel Ice Lake Xeon CPUs

Benchmarks were conducted using an Intel Ice Lake Xeon Platinum 8380 CPU on Ubuntu 20.04.2 LTS, testing PyTorch 1.9.0 and TensorFlow 2.5.0 across various batch sizes (1 to 128) and sequence lengths (8 to 512).

Eager vs. Graph Mode

  • Eager Mode (PyTorch, TensorFlow): The computation graph is discovered during execution. This offers flexibility but introduces runtime overhead and makes operator fusion (e.g., Convolution + ReLU) more difficult.
  • Graph Mode (TorchScript, TensorFlow Graph, Intel TensorFlow): The graph is known beforehand, allowing for pruning, operator fusion, and pre-planned memory allocations.

Key Findings on Scaling

  • Core Scaling: Increasing cores generally reduces latency, but not monotonically. There is a trade-off between workload size and allocated resources.
  • Inter-Socket Overhead: Using all cores on systems with multiple CPUs (multiple sockets) often introduces significant latency overhead due to inter-socket communication.
  • Allocator Impact: In eager mode, where memory must be managed dynamically, the choice of allocator (e.g., tcmalloc) has a more pronounced impact on performance than in graph mode, where resources can be reserved in advance.

Automatic Performance Tuning with Intel SigOpt

Because the search space for optimal settings is vast (combining core count, memory allocator, parallelism library, Transparent Huge Pages, and KMP block time), brute-force tuning is inefficient. Hugging Face utilized Intel SigOpt, which uses Bayesian optimization to find near-optimal configurations faster.

SigOpt Results

  • Efficiency: SigOpt provided performance very close to brute-force results, with the largest gap observed being 8.6%.
  • Parameter Importance: The number of cores is consistently the most critical parameter. However, for larger sequence lengths (e.g., 512), the choice of parallelism library (OpenMP vs. Intel OpenMP) becomes more significant relative to the memory allocator.
  • Resource Optimization: SigOpt identified that using fewer cores (e.g., 16 cores) can sometimes yield the best latency, allowing multiple model instances to run in parallel to improve overall throughput.

Conclusion

Optimizing BERT-like models for production on Intel Ice Lake Xeon CPUs requires a layered approach: starting with low-level hardware features, moving to framework-specific optimizations (like oneDNN), and finally tuning system-level knobs such as memory allocators and OpenMP implementations. These efforts are integrated into the Hugging Face Optimum library and the Infinity product to provide high-efficiency containerized inference solutions.

Sources