microsoft/Olive

Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.

What it solves

Olive simplifies the process of optimizing AI models for deployment. It addresses the problem of finding the best combination of optimization techniques to ensure a model runs efficiently on specific hardware (cloud or edge) while maintaining a required level of accuracy and latency.

How it works

Olive (ONNX LIVE) is a toolkit that takes a model and a target hardware specification. It composes a sequence of optimization techniques—such as quantization (e.g., GPTQ for int4 precision) and ONNX graph optimization—to produce an optimized ONNX model. It can automatically optimize popular architectures like Llama, Phi, Qwen, and Gemma out-of-the-box.

Who it’s for

Developers and ML engineers who need to deploy AI models to the ONNX Runtime for high-performance inference on various hardware targets, including NPUs and other edge devices.

Highlights

  • own a CLI for automatic optimization of popular model architectures.
  • Supports quantization algorithms and techniques for reducing model size and increasing speed.
  • Integrates with Hugging Face to acquire models directly.
  • Optimized for the ONNX Runtime to enable cross-platform, on-device inference.

Related

  • Project
    inisis/OnnxSlim

    A tool for slimming down ONNX models by reducing operators to improve inference speed and load times without sacrificing accuracy.

  • Project
    NVIDIA/Model-Optimizer

    NVIDIA Model Optimizer is an open‑source Python library for compressing and speeding up large AI models (LLMs, VLMs, diffusion). It supports PTQ, quant‑aware training, pruning, distillation, speculative decoding, and sparsity, works with Hugging Face, PyTorch, or ONNX inputs, and exports directly to TensorRT‑LLM, vLLM, SGLang, etc. Install via `pip install nvidia-modelopt[all]` or use NVIDIA containers. The repo includes docs, examples, and pre‑quantized checkpoints.

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    huggingface/optimum

    An extension of the Hugging Face ecosystem that provides optimization tools to train and run AI models with maximum efficiency on targeted hardware accelerators.

  • Project
    PINTO0309/PINTO_model_zoo

    A curated collection of pre‑converted deep‑learning models (image classification, object detection, etc.) in many deployment formats (TensorFlow Lite, ONNX, OpenVINO, CoreML, EdgeTPU, etc.). The repo provides the artefacts, licence info, and links to conversion tools, helping developers deploy models on diverse hardware without doing the conversion themselves.