Optimizing and Deploying Hugging Face Models with Optimum-Intel and OpenVINO GenAI

Hugging Face and Intel have introduced a workflow using Optimum-Intel and OpenVINO GenAI to optimize and deploy Transformers models on Intel hardware. This approach allows developers to move models from Python-based training environments to high-performance C++ or Python inference at the edge with minimal dependencies.

Efficient Edge Deployment via OpenVINO

OpenVINO is designed as a C++ AI inference solution, making it particularly suitable for edge and client-side deployments where minimizing software dependencies is critical. The introduction of the GenAI API further simplifies the integration of Large Language Models (LLMs) into applications, enhancing performance and reducing the complexity of deployment in environments where Python may not be ideal.

Exporting Transformers Models to OpenVINO IR

Through the Optimum-Intel project, Hugging Face Transformers models can be exported to the OpenVINO Intermediate Representation (IR) format. This process utilizes wrappers (such as OVModelForCausalLM) that mirror the 🤗 Transformers API.

Developers can export models using two primary methods:

  • Python API: Using the .from_pretrained() method with export=True.
  • Command Line Interface (CLI): Using the optimum-cli export openvino command.

Exporting a model creates a directory containing .xml and .bin IR model files, configuration files, and a tokenizer converted for the openvino-tokenizers library.

Model Optimization and Quantization

To reduce latency and memory footprint on resource-constrained devices, weight-only quantization is recommended. Optimum-Intel leverages the Neural Network Compression Framework (NNCF) to provide several optimization techniques:

  • Default Quantization: Models larger than one billion parameters are quantized to INT8 by default during export.
  • Advanced Quantization: 4-bit integer weight-only quantization provides a better accuracy-performance trade-off. Recommended techniques for models like Llama-3.1-8B include stacking AWQ (Activation-aware Weight Quantization), quantization scale estimation, and mixed-precision INT4/INT8 quantization using a calibration dataset.

Accuracy Impact of Quantization

Quantization typically introduces some degradation in accuracy. Using the Word Perplexity (PPL) metric on the Wikitext dataset, the impact on meta-llama/Meta-Llama-3.1-8B is as follows:

Model PPL PyTorch FP32 OpenVINO INT8 OpenVINO INT4
Meta-Llama-3.1-8B 7.3366 7.3463 7.8288

Deployment with OpenVINO GenAI API

Once a model is converted and optimized, the LLMPipeline class in OpenVINO GenAI enables deployment with minimal dependencies in both Python and C++.

Python Deployment

Deployment in Python requires only the openvino-genai package. The LLMPipeline handles model loading and text generation via a GenerationConfig object to control parameters like max_new_tokens.

C++ Deployment

The C++ API is designed for seamless migration from the 🤗 Transformers API. It allows developers to specify the hardware device (e.g., "CPU" or "GPU") and use ov::genai::LLMPipeline for generation.

Advanced Generation Features

LLMPipeline supports several high-level features to improve user experience and performance:

  • Custom Decoding: Support for algorithms such as Beam Search.
  • Interactive Chat: The start_chat() and finish_chat() methods allow for interactive scenarios. These methods utilize the KV cache of previous chat history to reduce prompt processing time.
  • Streaming: A streamer function can be implemented to print tokens as they are generated, rather than waiting for the full sequence.

Technical Requirements

For implementation, the following package versions are specified:

  • transformers: 4.44
  • openvino: 24.3
  • openvino-tokenizers: 24.3
  • optimum-intel: 1.20
  • lm-eval: 0.4.3
  • openvino-genai: 24.3

Sources