openvinotoolkit/nncf

Neural Network Compression Framework (NNCF)

What it is – NNCF is a Python library that adds model‑size‑reduction and inference‑speed‑up techniques to neural networks. It works with models built in PyTorch, TorchFX, ONNX and OpenVINO, letting you apply post‑training or training‑time compression with only a few lines of code.

Key capabilities

Category	Algorithms (as listed in the README)	Supported back‑ends
Post‑training	• 8‑bit Quantization
• Weights Compression
• Activation Sparsity (experimental)	OpenVINO, PyTorch, TorchFX (experimental), ONNX
Training‑time	• Quantization‑Aware Training (QAT)
• Weight‑Only QAT with LoRA/NLS
• Pruning	PyTorch only
Other features	• Automatic graph transformation,
• GPU‑accelerated layers for fine‑tuning,
• Distributed training support,
• Integration patches for HuggingFace‑Transformers,
• Export to ONNX, SavedModel, or Frozen Graph for OpenVINO runtime

Typical use‑cases

• Deploying a PyTorch or ONNX model on edge devices where memory and latency are limited.
• Reducing the size of large language models (e.g., BERT, LLMs) while keeping accuracy acceptable.
• Preparing models for the OpenVINO inference engine, especially for CPUs, GPUs, or VPUs.
• Researchers who want to experiment with pruning or quantization‑aware training without writing low‑level code.

How to get started

# Install the package from PyPI (requires Python 3.10+)
pip install nncf

The library can also be built from source if you need the latest development version.

Basic workflow (post‑training quantization example)

import nncf, torch, openvino as ov
from torchvision import datasets, transforms

# 1️⃣ Load a model (OpenVINO, PyTorch, TorchFX or ONNX)
model = ov.Core().read_model("/model_path")   # OpenVINO example

# 2️⃣ Prepare a small calibration dataset (~300 samples)
val_ds = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
loader = torch.utils.data.DataLoader(val_ds, batch_size=1)

# 3️⃣ Wrap the loader in an NNCF Dataset
calib = nncf.Dataset(loader, lambda d: d[0])

# 4️⃣ Run the quantization pipeline
quantized = nncf.quantize(model, calib)

The same pattern works for PyTorch, TorchFX and ONNX back‑ends; the README provides concrete snippets for each.

Learning resources

• Official docs – https://docs.openvino.ai/nncf (full user guide, API reference, and algorithm details).
• Model Zoo – a curated list of models with reported compression results: docs/ModelZoo.md.
• Jupyter notebooks – ready‑to‑run tutorials covering BERT quantization, segmentation models, YOLOv11, LLM weight compression, etc. (links in the Demos, Tutorials and Samples section).
• API reference – https://openvinotoolkit.github.io/nncf/autoapi/nncf/ (auto‑generated docs for classes and functions).

Who should consider NNCF?

• Engineers deploying AI on resource‑constrained hardware (edge, IoT, mobile).
• Teams using the OpenVINO toolkit who need a seamless way to shrink models before inference.
• Researchers exploring quantization‑aware training or pruning in PyTorch.

---

All information above is taken directly from the repository’s README; no additional features are inferred.