Intel and Hugging Face Partnership for Machine Learning Hardware Acceleration

Intel and Hugging Face have partnered to democratize machine learning hardware acceleration by integrating Intel's AI software and Xeon hardware with the Hugging Face ecosystem. This collaboration centers on the Optimum Intel open-source library, which simplifies the process of optimizing Transformer models for better performance, scale, and productivity on Intel platforms.

Streamlining Transformer Acceleration with Optimum Intel

Optimum Intel is a specialized component of the broader Hugging Face Optimum library, designed to reduce the complexity of latency optimization for machine learning practitioners. It builds upon the Intel Neural Compressor (INC), an open-source library that provides unified interfaces for network compression technologies across multiple deep learning frameworks.

Key capabilities provided by the Intel Neural Compressor include:

  • Quantization: Reducing the bit width of model parameters (e.g., from 32-bit floating-point to 8-bit integers) to decrease memory and compute requirements.
  • Pruning: Zeroing or removing model parameters that have minimal impact on the predicted outcome.
  • Knowledge Distillation: Techniques to compress models while maintaining performance.

Optimum Intel allows both beginners and experts to apply these state-of-the-art optimization techniques using ready-made scripts or minimal code changes.

Performance Benchmarks and Hardware Support

The partnership leverages Intel's foundation for accelerated AI, specifically the Intel Xeon Scalable CPU platform and its associated hardware-optimized AI software tools.

Previous collaborative efforts between the two organizations have demonstrated significant performance gains:

  • Inference Latency: Single-digit millisecond latency was achieved for DistilBERT on Intel Xeon Ice Lake CPUs.
  • Training Efficiency: Support for Habana Gaudi accelerators provides up to 40% better price-performance compared to GPUs.

Case Study: Post-Training Quantization of DistilBERT

To demonstrate the utility of Optimum Intel, a case study was performed on a DistilBERT model fine-tuned for shoe review classification. The process utilized post-training dynamic quantization to shrink the model's memory and compute footprint.

Technical Implementation

Using the INCQuantizer from the optimum.intel.neural_compressor module, the model was quantized with a configuration that allowed for a maximum accuracy drop of 5% across 10 trials. The process replaced standard Linear layers with DynamicQuantizedLinear layers.

Results

The quantization of 38 Linear and 2 Embedding operators resulted in the following performance shifts:

  • Execution Speed: The evaluation step of the quantized model was 1.34x faster than the original model.
  • Accuracy: The accuracy decreased from 0.574 to 0.546.
  • Latency: The duration of the evaluation process dropped from 13.1534 seconds to 9.7695 seconds.

Future Outlook

The partnership aims to continue expanding the capabilities of the Optimum Intel library, moving beyond post-training quantization to include more advanced pruning and quantization techniques to ensure peak efficiency for users on Intel Xeon CPUs and Intel AI libraries.

Sources