Optimum 1.2 Inference Release Adds ONNX Runtime Accelerated Pipelines

TL;DR

Optimum 1.2 introduces inference support for Hugging Face Transformers pipelines using ONNX Runtime, allowing users to replace standard AutoModel classes with ORTModel equivalents, apply graph optimizations and dynamic quantization, and achieve up to 2× latency reduction while preserving >99% of original accuracy.

What is Optimum?

Optimum is an open‑source library that extends Hugging Face Transformers with a unified API for performance optimizations on accelerated hardware. It provides tools for accelerated training, quantization, graph optimization, and now inference, supporting hardware such as Graphcore IPU, Habana Gaudi, and ONNX Runtime.

New Inference and Pipeline Features in Optimum 1.2

  • API Compatibility – ORTModelForXxx classes are drop‑in replacements for AutoModelForXxx. Example:
    from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForQuestionAnswering

model = ORTModelForQuestionAnswering.from_pretrained("optimum/roberta-base-squad2")
tokenizer = AutoTokenizer.from_pretrained("deepset/roberta-base-squad2")
qa = pipeline("question-answering", model=model, tokenizer=tokenizer)
  • ONNX Runtime Integration – Models are exported to ONNX and executed with ONNX Runtime, gaining operator fusion, constant folding, and hardware‑specific kernels.
  • Optimization & Quantization – ORTOptimizer applies graph‑level optimizations; ORTQuantizer performs dynamic quantization (e.g., AVX‑512 VNNI) to shrink model size and improve latency.
  • Hub Compatibility – Optimized checkpoints can be pushed to and pulled from the Hugging Face Hub, enabling community sharing of accelerated models.
  • Pipeline Safety Layer – optimum.pipelines.pipeline validates task‑model compatibility and raises errors for unsupported configurations.

End‑to‑End Tutorial: Accelerating RoBERTa for Question‑Answering

The blog post walks through a six‑step workflow on an AWS m5.xlarge instance:

  1. Install Optimum with ONNX Runtime
    pip install "optimum[onnxruntime]==1.2.0"
  2. Convert a Transformers model to ONNX
    from optimum.onnxruntime import ORTModelForQuestionAnswering
model = ORTModelForQuestionAnswering.from_pretrained(
    "deepset/roberta-base-squad2", from_transformers=True)
model.save_pretrained("onnx")
  3. Apply Graph Optimizations
    from optimum.onnxruntime import ORTOptimizer, OptimizationConfig
optimizer = ORTOptimizer.from_pretrained("deepset/roberta-base-squad2", feature="question-answering")
optimizer.export(
   
   
    optimization_config=OptimizationConfig(optimization_level=99),
)
  4. Dynamic Quantization
    from optimum.onnxruntime import ORTQuantizer, AutoQuantizationConfig
quantizer = ORTQuantizer.from_pretrained("deepset/roberta-base-squad2", feature="question-answering")
qconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=True)
quantizer.export(
   
   
    quantization_config=qconfig,
)
  5. Run Inference with Transformers Pipelines
    from transformers import pipeline, AutoTokenizer
from optimum.onnxruntime import ORTModelForQuestionAnswering
tokenizer = AutoTokenizer.from_pretrained("onnx")
model = ORTModelForQuestionAnswering.from_pretrained("onnx", file_name="model-quantized.onnx")
qa = pipeline("question-answering", model=model, tokenizer=tokenizer)
  6. Evaluate Accuracy and Latency
    • Accuracy on SQuAD‑v2: vanilla 82.15 F1, optimized 82.15 F1, quantized 81.83 F1 (≈99.6 % of original).
    • Latency on a 2‑core CPU: vanilla ≈ 117 ms, optimized + quantized ≈ 65 ms (≈2× speed‑up).

Current Limitations

  • Model size – Only models < 2 GB can be loaded from the Hub.
  • Seq2Seq support – Tasks such as summarization (e.g., T5) are not yet available.
  • Past key values – Causal language models (e.g., GPT‑2) do not yet use cached attention states.
  • Local caching – Optimized ONNX files are not cached locally after download.

Frequently Asked Questions

  • Supported tasks – Feature extraction, text classification, token classification, question answering, zero‑shot classification, and text generation.
  • Supported models – Any model exportable via transformers.onnx, including BERT, ALBERT, RoBERTa, XLM‑R, DistilBERT, GPT‑2, etc.
  • Supported runtimes – Currently ONNX Runtime; additional runtimes (TensorRT, AWS‑Neuron) are planned.
  • GPU usage – Install optimum[onnxruntime-gpu] to enable GPU providers automatically.
  • Loading optimized models – Use the ORTModelForXXX.from_pretrained(..., file_name="model‑optimized.onnx") pattern.

Roadmap and Future Work

Optimum aims to become the reference toolkit for transformer acceleration. Upcoming milestones include:

  • Adding speech (Wav2Vec 2.0) and vision (ViT) model support.
  • Integrating OrtValue and IOBinding for lower‑level performance gains.
  • Providing easier evaluation utilities for accelerated models.
  • Expanding runtime support to TensorRT, AWS‑Neuron, and other providers.

If you are interested in contributing or learning more, the Optimum repository, Hugging Face forum, and the author’s social channels are open for discussion.

Sources