optimum: a hardware-optimization toolkit for maximizing the efficiency of training and inference across diverse AI accelerators

What it solves

Optimum simplifies the process of training and running AI models on specific hardware accelerators, ensuring they achieve maximum efficiency without requiring the user to manually handle complex hardware-specific optimizations. It extends the functionality of libraries like Transformers, Diffusers, TIMM, and Sentence-Transformers.

How it works

Optimum acts as an optimization layer that provides tools to export, quantize, and run models across various ecosystems. It offers specialized wrappers and integrations for different hardware backends, including:

• Inference Engines: Support for ONNX Runtime, OpenVINO, ExecuTorch, NVIDIA TensorRT-LLM, and AWS Inferentia.
• Training Wrappers: Specialized wrappers around the Transformers Trainer to enable accelerated training on hardware like Intel Gaudi (HPU) and AWS Trainium.
• Quantization: Tools like Quanto for PyTorch quantization via API or command line.

Who it’s for

Developers and ML engineers who need to deploy AI models to production on targeted hardware (edge devices, GPUs, NPUs, and specialized AI accelerators) and those who want to accelerate their training pipelines.

Highlights

• Broad Hardware Support: Integrates with NVIDIA, Intel (OpenVINO, Gaudi), AMD, AWS (Trainium, Inferentia), and FuriosaAI.
• Seamless Export: Enables easy export of models to formats like ONNX and ExecuTorch for on-device inference.
• Unified Interface: Provides a consistent way to optimize and run models from the Hugging Face ecosystem using Python APIs or a command-line interface.
• Accelerated Training: Simplifies the use of high-performance hardware for model training and fine-tuning.