NVIDIA/Model-Optimizer

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

NVIDIA Model Optimizer – What It Is

NVIDIA Model Optimizer (ModelOpt) is an open‑source Python library that helps you make large AI models—especially large language models (LLMs) and vision‑language models (VLMs)—run faster and use less memory. It does this by applying a suite of model‑optimization techniques such as:

Technique What it does Typical benefit
Post‑Training Quantization (PTQ) Converts weights and activations to lower‑precision formats (e.g., FP8, NVFP4, INT4) after training. 2‑4× smaller checkpoints, up to ~2× higher inference throughput.
Quantization‑Aware Training / Distillation (QAD) Fine‑tunes a model while it is already quantized, or teaches a smaller model to mimic a larger one. Recovers accuracy lost from aggressive quantization while keeping the speed gains.
Pruning Removes weights that contribute little to the output. Reduces parameter count and memory, often with minimal accuracy loss.
Distillation Trains a compact “student” model to reproduce the behavior of a big “teacher” model. Smaller, faster models that retain most of the teacher’s quality.
Speculative Decoding Trains a draft model that predicts several tokens ahead, then corrects them with the full model. Lowers latency for token‑by‑token generation.
Sparsity Stores only non‑zero weights and their locations. Further memory savings and compute reduction.

How It Fits Into an AI Workflow

  1. Input – You start with a model from Hugging Face, a native PyTorch checkpoint, or an ONNX file.
  2. Optimize – Using ModelOpt’s Python API (or the provided CLI), you compose any combination of the techniques above. The library also integrates with NVIDIA’s Megatron‑Bridge, Megatron‑LM, and Hugging Face Accelerate to make training‑time optimizations easier.
  3. Export – The resulting optimized checkpoint can be exported directly for downstream inference engines such as TensorRT‑LLM, vLLM, SGLang, or TensorRT. The export API works for both transformer and diffusion (diffusers) models.

Who Uses It?

  • NVIDIA’s own teams (e.g., Nemotron‑3, DeepSeek‑R1) to produce quantized checkpoints that are publicly shared on Hugging Face.
  • Enterprises like Domyn and Bielik.AI that have published case studies showing 30‑50% speedups and sizable model‑size reductions.
  • Researchers and developers who need a reproducible, open‑source way to prepare models for production GPUs (including the new Blackwell RTX 50 series).

Getting Started

# Install the stable release from PyPI (includes all optional deps)
pip install -U nvidia-modelopt[all]

You can also clone the repo and install in editable mode for the latest features, or pull an NVIDIA container image that already contains ModelOpt.

Documentation & Resources

When to Use Model Optimizer

  • You have a large transformer model you want to run on NVIDIA GPUs with limited memory.
  • You need to ship a model to a production inference service (TensorRT‑LLM, vLLM, SGLang) and want the best throughput‑vs‑accuracy trade‑off.
  • You are experimenting with newer low‑precision formats like NVFP4 or FP8 and need tooling to search per‑layer precision assignments.

Limitations & Policy

  • The project is still pre‑1.0 (0.x releases). Deprecations are announced in the changelog and have a ~1‑month migration window.
  • ModelOpt pulls in several third‑party OSS dependencies; you must review their licenses.

TL;DR: NVIDIA Model Optimizer is a real, actively maintained open‑source library that lets you compress, prune, quantize, and otherwise speed up large AI models, with seamless export to NVIDIA’s inference runtimes.

Related

  • Project
    NVIDIA/TensorRT

    A high-performance inference optimizer and runtime from NVIDIA that accelerates AI model deployment and throughput on NVIDIA GPUs.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    huggingface/optimum

    An extension of the Hugging Face ecosystem that provides optimization tools to train and run AI models with maximum efficiency on targeted hardware accelerators.

  • Project
    inisis/OnnxSlim

    A tool for slimming down ONNX models by reducing operators to improve inference speed and load times without sacrificing accuracy.

  • Project
    microsoft/Olive

    An AI model optimization toolkit that composes the best optimization techniques to create efficient ONNX models for cloud or edge inference based on hardware and performance constraints.