NVIDIA/Model-Optimizer

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

NVIDIA Model Optimizer (ModelOpt)

What it is – An open‑source Python library from NVIDIA that bundles a set of state‑of‑the‑art model‑optimization techniques (quantization, pruning, NAS, distillation, speculative decoding, sparsity) and provides a unified API to apply them to Hugging Face, PyTorch, or ONNX models. The output is an optimized checkpoint ready for fast inference with NVIDIA’s deployment stacks (TensorRT‑LLM, TensorRT, vLLM, SGLang, etc.).

Key capabilities

Technique What it does Typical benefit
Post‑Training Quantization (PTQ) Reduces model size 2‑4×, lowers memory bandwidth and latency Faster inference with little accuracy loss
Quantization‑Aware Training / Distillation (QAT) Fine‑tunes a quantized model to recover accuracy Enables aggressive low‑precision formats (FP8, NVFP4) while keeping quality
Pruning Removes unimportant weights or entire channels Smaller memory footprint and higher throughput
Distillation Trains a smaller “student” model to mimic a larger “teacher” Deployable models that retain most of the teacher’s performance
Speculative Decoding Trains a draft model that predicts extra tokens ahead of time Reduces token‑generation latency for LLMs
Sparsity Stores only non‑zero parameters and their locations Further memory/computation savings

Typical workflow

  1. Input – Provide a model in Hugging Face, PyTorch, or ONNX format.
  2. Compose – Use the Python API (or the provided examples) to chain techniques, e.g., prune → distill → QAT → PTQ.
  3. Export – Generate a quantized checkpoint that can be loaded directly by downstream runtimes such as TensorRT‑LLM, vLLM, SGLang, or NVIDIA’s Megatron‑Bridge.

Installation

# Stable release from PyPI (includes all optional deps)
pip install -U nvidia-modelopt[all]

For development or the latest features:

git clone https://github.com/NVIDIA/Model-Optimizer.git
cd Model-Optimizer
pip install -e .[dev]

Container images with ModelOpt pre‑installed are also available on nvcr.io.

Documentation & resources

  • Full docs: https://nvidia.github.io/Model-Optimizer
  • Quick‑start tutorials for each technique are under examples/ (e.g., hf_ptq, pruning, speculative_decoding).
  • Pre‑quantized checkpoints for popular LLMs/VLMs are hosted on Hugging Face (see the collection link in the README).
  • Roadmap, benchmarks, and changelog are linked from the repo.

Who should use it

  • Engineers deploying large language or vision models on NVIDIA GPUs who need to squeeze out extra throughput or reduce memory usage.
  • Researchers experimenting with low‑precision formats (FP8, NVFP4) or model‑compression pipelines.
  • Teams that already use NVIDIA’s inference stack (TensorRT, TensorRT‑LLM, Megatron‑Bridge) and want a single library to prepare models.

License – Apache 2.0 (see LICENSE).

Citation – A ready‑to‑copy BibTeX entry is provided for academic use.

All information above is taken directly from the repository’s README; no external assumptions have been added.

Related

  • Project
    vllm-project/llm-compressor

    A library for quantizing and compressing large language models (weights, activations, KV‑cache, etc.) so they run efficiently with the vLLM inference engine. Supports many algorithms (GPTQ, AWQ, AutoRound, REAP pruning, etc.) and formats (FP8, NVFP4, INT4, mixed‑precision). Provides one‑line API, DDP + disk‑offloading for multi‑terabyte models, and pre‑quantized checkpoints for popular LLMs.

  • Project
    NVIDIA/TensorRT

    NVIDIA TensorRT is a high-performance deep learning inference library and SDK that accelerates AI model deployment on NVIDIA GPUs.

  • Project
    huggingface/optimum

    An extension of the Hugging Face ecosystem that provides optimization tools to train and run AI models with maximum efficiency on targeted hardware accelerators.

  • Project
    microsoft/Olive

    An AI model optimization toolkit that composes the best optimization techniques to create efficient ONNX models for cloud or edge inference based on hardware and performance constraints.

  • Dispatch
    Optimum-NVIDIA Release Notes

    Hugging Face has released Optimum-NVIDIA, an inference library that accelerates LLM inference on NVIDIA platforms by up to 28x using FP8 quantization and TensorRT-LLM.