microsoft/Olive

Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.

What it solves

Olive simplifies the process of optimizing AI models for deployment. It addresses the problem of finding the best combination of optimization techniques to ensure a model runs efficiently on specific hardware (cloud or edge) while maintaining a required level of accuracy and latency.

How it works

Olive (ONNX LIVE) is a toolkit that takes a model and a target hardware specification. It composes a sequence of optimization techniques—such as quantization (e.g., GPTQ for int4 precision) and ONNX graph optimization—to produce an optimized ONNX model. It can automatically optimize popular architectures like Llama, Phi, Qwen, and Gemma out-of-the-box.

Who it’s for

Developers and ML engineers who need to deploy AI models to the ONNX Runtime for high-performance inference on various hardware targets, including NPUs and other edge devices.

Highlights

  • own a CLI for automatic optimization of popular model architectures.
  • Supports quantization algorithms and techniques for reducing model size and increasing speed.
  • Integrates with Hugging Face to acquire models directly.
  • Optimized for the ONNX Runtime to enable cross-platform, on-device inference.

Related

  • Project
    inisis/OnnxSlim

    A tool for simplifying and slimming ONNX models to reduce operator count and improve inference speed while maintaining accuracy.

  • Project
    NVIDIA/Model-Optimizer

    NVIDIA Model Optimizer (ModelOpt) is an open‑source Python library that lets you apply quantization, pruning, NAS, distillation, speculative decoding, and sparsity to Hugging Face, PyTorch, or ONNX models and export them for fast inference with NVIDIA runtimes like TensorRT‑LLM, vLLM, and SGLang.

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    huggingface/optimum

    An extension of the Hugging Face ecosystem that provides optimization tools to train and run AI models with maximum efficiency on targeted hardware accelerators.

  • Project
    PINTO0309/PINTO_model_zoo

    A comprehensive repository of AI models inter-converted and quantized for various frameworks like TensorFlow Lite, ONNX, and CoreML to simplify edge deployment.