Hugging Face Optimum Intel and OpenVINO Integration

Hugging Face has integrated Intel OpenVINO into the Optimum Intel library, allowing users to perform accelerated inference and quantize Transformer models for deployment on a wide range of Intel processors. This integration simplifies the process of reducing model size and prediction latency using the OpenVINO Runtime and the Neural Network Compression Framework (NNCF).

OpenVINO Integration and Model Support

The first release of this integration is based on OpenVINO 2022.2. It enables inference for a large number of PyTorch models via the OVModels class.

Supported Model Types

  • Encoder Models: Post-training static quantization and quantization-aware training are supported for many encoder models, such as BERT and DistilBERT. Additional encoder models are expected in future OpenVINO releases.
  • Encoder-Decoder Models: Quantization for these models is not currently enabled, but support is planned for integration with the next OpenVINO release.

Model Quantization with NNCF

Quantization reduces the bit width of model parameters to lower memory and compute requirements. This allows for faster matrix multiplication through integer arithmetic and reduced memory usage during inference.

The Quantization Process

Using the OVQuantizer and the OpenVINO Neural Network Compression Framework (NNCF), users can perform post-training static quantization. This process involves a calibration step where a small subset of data (e.g., 300 samples) is fed through the network to compute quantized activation parameters.

Once quantized, the model is exported to the OpenVINO Intermediate Representation (IR) format, consisting of an XML file for network topology and a binary file for weights. This format allows the model to run on any target Intel device.

Performance Gains: Vision Transformer (ViT) Case Study

To demonstrate the effectiveness of the integration, Hugging Face applied post-training static quantization to a Vision Transformer (ViT) model fine-tuned on the food101 dataset. The results showed significant efficiency gains with minimal impact on accuracy.

Efficiency and Latency Benchmarks

  • Memory Reduction: The model size decreased by 3.8x, dropping from 344MB to 90MB.
  • Latency Improvement: Inference latency improved by 2.4x, decreasing from 98ms to 41ms per sample (based on 5,050 image predictions).
  • Accuracy: Both the original and quantized models maintained an accuracy of 87.6 on a 20% subset of the evaluation dataset.

Implementation Note

Because the model is compiled immediately before the first inference, the initial prediction will have inflated latency. Users are advised to perform at least one warmup prediction before conducting benchmarks.

Sources