Hugging Face Optimum Release

Hugging Face has launched Optimum, an open-source library designed to democratize machine learning production performance by providing a toolkit for optimizing Transformer models for specific hardware. The library aims to abstract the complexity of model acceleration, enabling engineers to maximize efficiency during training and inference without requiring deep expertise in hardware-specific software tooling.

Solving the Complexity of Scaling Transformers

Scaling Transformer models for production is challenging because optimal performance requires acceleration techniques that are specifically compatible with the target hardware. This involves navigating a complex compatibility matrix of software tooling, hardware features, and optimized kernels that must align with the operators on silicon and the specific neural network graph of the model architecture.

While the Transformers library simplified the use of state-of-the-art models by abstracting framework and architecture complexities, Optimum abstracts the complexity of model acceleration on hardware platforms to make these features accessible to a broader range of machine learning engineers.

Hardware-Specific Optimization and Quantization

Optimum focuses on achieving maximum efficiency by working directly with hardware partners to enable, test, and maintain acceleration techniques. A primary example of this integration is the collaboration with Intel to optimize models for Intel Xeon CPUs.

The Challenge of Quantization

Quantization is a widely used technique to reduce the compute power and cost of deploying Transformer-based models (such as BERT, ViT, and Speech2Text). However, implementing quantization is typically difficult due to several factors:

• Model Editing: Certain operations must be replaced with quantized counterparts, and new quantization/dequantization nodes must be inserted. In eager-mode frameworks like PyTorch, this often requires modifying the model implementation itself.
• Parameter Tuning: Engineers must determine the correct observers for range calibration, select appropriate quantization schemes, and identify supported data types (e.g., int8, uint8, int16) for the target device.
• Accuracy Trade-offs: There is a constant need to balance the reduction in model size and increase in speed against acceptable losses in accuracy.
• Export Complexity: The final quantized model must be exported specifically for the target device.

Integration with Intel Neural Compressor

To address these challenges on Intel hardware, Optimum integrates with the Intel® Neural Compressor (formerly Low Precision Optimization Tool or LPOT). This open-source Python library allows users to deploy low-precision inference solutions using recipes for post-training quantization, quantization-aware training, and dynamic quantization. Users specify their tuning parameters, objectives, and performance criteria via a configuration YAML file, which can be stored locally or hosted on the Hugging Face Model Hub.

Future Goals and Ecosystem Impact

Optimum is designed to be a collaborative effort between Hugging Face and various hardware partners to deliver hardware-specific optimized model configurations and artifacts. These optimized models will be made available via the Hugging Face Model Hub.

By providing these tools, Hugging Face aims to reduce the aggregate energy spent on production machine learning workloads and accelerate the adoption of Transformers at scale for organizations of all sizes, not just large technology companies.