Hugging Face Infinity CPU Performance Case Study
Hugging Face Infinity is a containerized inference solution designed to reduce latency and increase throughput for Transformer models deployed on CPU infrastructure. By optimizing models for specific hardware architectures, it enables real-time use cases and reduces infrastructure costs for large-scale deployments.
Architecture of Hugging Face Infinity
Infinity consists of two primary components that work together to provide a hardware-accelerated inference pipeline:
- Infinity Container: A Docker container that provides a hardware-optimized inference solution and exposes an HTTP
/predictendpoint. - Infinity Multiverse: A model optimization service that optimizes a Hugging Face Transformer model for the target hardware architecture to ensure compatibility with the Infinity Container.
Each Infinity Container is designed to serve a single model and a single task. Supported tasks include token classification, sequence classification, ranking, and feature extraction/document embedding.
Performance Benchmarks on Intel Ice Lake
In a case study using Amazon EC2 C6i instances (powered by 3rd generation Intel Xeon Scalable processors with Ice Lake architecture), Hugging Face benchmarked a DistilBERT model for sequence classification.
End-to-End Measurement
Unlike benchmarks that only measure model execution, Infinity's performance is measured as an end-to-end pipeline, encompassing pre-processing, prediction, and post-processing.
Key Results
Infinity running on Ice Lake optimized containers achieved the following improvements over other configurations:
- Versus Vanilla Transformers: Up to 800% better latency and throughput on Ice Lake hardware.
- Versus Cascade Lake: Up to 34% better latency and throughput compared to instances based on the Cascade Lake architecture.
Throughput Comparison
When running on two physical cores with a batch size of one, Infinity showed significant throughput gains over vanilla Transformers across all sequence lengths:
| Sequence Length | Infinity | Transformers | Improvement |
|---|---|---|---|
| 8 | 248 req/sec | 49 req/sec | +506% |
| 16 | 212 req/sec | 16 req/sec | +424% |
| 32 | 150 req/sec | 40 req/sec | +375% |
| 64 | 97 req/sec | 28 req/sec | +346% |
| 128 | 55 req/sec | 18 req/sec | +305% |
| 256 | 27 req/sec | 9 req/sec | +300% |
| 384 | 17 req/sec | 5 req/sec | +340% |
| 512 | 12 req/sec | 4 req/sec | +300% |
Latency Metrics
For sequence lengths up to 64 tokens, Infinity can deliver latency as low as 1-4ms. The benchmarks also demonstrated that Infinity maintains consistent performance with minimal deviation between p95, p99, and maximum latency (p100).
Deployment Implications
The ability to optimize for either throughput or latency allows businesses to either reduce infrastructure costs for the same workload or enable real-time AI applications that were previously impractical on CPU-based infrastructure.
Note: As of December 2022, Hugging Face Infinity is no longer offered as a commercial inference solution. Hugging Face now recommends using Inference Endpoints for managed infrastructure, or the Optimum Intel and Optimum ONNX Runtime libraries for open-source hardware optimization.