Scaling-up BERT Inference on CPU (Part 1)

Hugging Face has detailed a methodology for scaling BERT-like model inference on modern CPUs, demonstrating that maximum throughput is achieved by deploying multiple independent model instances bound to specific physical CPU cores. This approach avoids the overhead of Simultaneous Multi-Threading (SMT) and cross-socket communication, allowing for nearly linear scalability in throughput.

Out-of-the-Box Framework Performance

Initial benchmarks conducted on an AWS c5.metal instance (Intel Xeon Platinum 8275 CPU, 48 cores/96 threads) show that PyTorch (1.8.1) generally outperforms Google TensorFlow (2.4.1) for BERT inference out-of-the-box across various configurations. This performance difference is attributed to the underlying execution technologies: PyTorch utilizes OpenMP and Intel MKL (oneDNN), while TensorFlow relies on Eigen and its own threading implementation.

CPU Architecture and Thread Affinity

To optimize CPU-bound tasks like BERT inference, which primarily consist of general matrix multiplications (GEMMs), specific hardware considerations are required:

Simultaneous Multi-Threading (SMT)

SMT (or Hyper-Threading) allows two software threads to share a single physical core. However, because BERT inference is CPU-bound, logical cores compete for the same execution resources, meaning SMT provides no performance benefit and should be avoided in favor of using only physical cores.

NUMA and Socket Management

Modern multi-socket servers use Non-Uniform Memory Access (NUMA), where each CPU socket manages its own subset of memory. To prevent performance degradation caused by cross-socket communication overhead, processes must be bound to specific cores and memory pools using tools like numactl.

Key Configuration Strategy:

  • Thread Affinity: Binding a process to a specific set of physical cores (e.g., numactl -C 0-47).
  • Memory Allocation: Ensuring memory is allocated on the socket closest to the cores performing the computation (e.g., numactl -m 0,1).

Core Count Scaling vs. Multi-Stream Inference

There are two primary ways to scale resources for inference, each with different impacts on latency and throughput.

Core Count Scaling (Strong Scaling)

This involves increasing the number of cores assigned to a single task to reduce latency. The effectiveness of this approach depends on the problem size:

  • Small to Medium Problems: Using a single socket often yields the best performance.
  • Large Problems: The overhead of cross-socket communication is offset by the computational cost, making the use of all available cores across both sockets beneficial.

Multi-Stream Inference (Instance Parallelism)

Instead of assigning more cores to one instance, this method allocates multiple independent model instances, each bound to a non-overlapping subset of physical cores.

For example, running four independent instances each bound to 12 cores results in a slight increase in individual latency compared to a single instance with 24 cores, but increases overall throughput by 4x. This allows for "smart dispatching," where requests are routed to specific instances tuned for different sequence lengths (e.g., 8 cores for short sequences, 24 cores for long sequences).

Batch Size Scaling and Throughput

Batch Size Scaling involves splitting a global batch $B$ across $N$ instances, where each instance processes a sub-batch of $B/N$ on $C/N$ cores.

Latency and Throughput Trade-offs

  • Latency: The overall latency for a batch is determined by the slowest instance in the pool. The optimal number of instances to minimize latency varies by problem size; for a batch of 8 and sequence length of 128, 4 instances (each with a batch of 2 and 12 cores) provided the best results.
  • Throughput: Throughput scales almost linearly as more instances are added, provided the workload per instance is reduced proportionally. This indicates optimal hardware utilization.

Summary of Hardware Optimization Impact

By correctly tuning thread affinity and instance allocation, organizations can significantly reduce infrastructure costs. Hugging Face notes that for certain workloads, moving from a 48-core machine ($4.848/h) to an 8-core instance ($0.808/h) can result in a 6x cost reduction if the problem size does not require the additional cores for optimal latency.

Sources