Accelerating Hugging Face Models with ONNX Runtime

Hugging Face and ONNX Runtime have integrated to accelerate over 130,000 models on the Hugging Face Hub, significantly improving inference performance across a wide range of machine learning architectures. This integration allows users to leverage cross-platform acceleration to reduce latency and optimize deployment for both large language models (LLMs) and cloud models.

Performance Gains in Model Inference

ONNX Runtime provides substantial latency improvements for Hugging Face models. For instance, the whisper-tiny model demonstrates an average latency per inference gain of up to 74.30% when using ONNX Runtime compared to PyTorch.

Supported Model Architectures

ONNX Runtime supports over 90 Hugging Face model architectures. This support extends to the most widely used models on the Hub, including the 11 most popular architectures based on the number of models uploaded:

Model Architecture	Approximate No. of Models
BERT	28,180
GPT2	14,060
DistilBERT	11,540
RoBERTa	10,800
T5	10,450
Wav2Vec2	6,560
Stable-Diffusion	5,880
XLM-RoBERTa	5,100
Whisper	4,400
BART	3,590
Marian	2,840

Technical Overview of ONNX Runtime

ONNX Runtime is a cross-platform machine learning tool designed to accelerate a variety of models, particularly those that have ONNX support. By collaborating with Hugging Face, ONNX Runtime ensures that the most popular model architectures on the platform are optimized for high-performance execution.