openvinotoolkit/nncf

Neural Network Compression Framework for enhanced OpenVINO™ inference

What it solves

NNCF optimizes the inference of neural networks by reducing their size and complexity, allowing them to run faster and use fewer resources with minimal loss in accuracy. It provides tools to compress models from PyTorch, ONNX, and OpenVINO backends.

How it works

NNCF uses a variety of compression algorithms categorized into two main types:

  • Post-Training Compression: These algorithms are applied after a model has been trained, requiring only a small calibration dataset to optimize the model. This includes Post-Training Quantization (PTQ) and Weights Compression.
  • Training-Time Compression: These algorithms are integrated into the training or fine-tuning process. This includes Quantization Aware Training (QAT), Pruning, and Weight-Only Quantization Aware Training using LoRA and NLS.

Who it’s for

It is designed for developers and ML engineers who need to deploy neural networks to production environments where inference speed and memory efficiency are critical, particularly when using the OpenVINO toolkit.

Highlights

  • Multi-backend support: Works with PyTorch, TorchFX, ONNX, and OpenVINO.
  • Comprehensive toolset: Offers both post-training and quantization-aware training methods.
  • GPU acceleration: Includes GPU-accelerated layers for faster fine-tuning of compressed models.
  • Integration: Provides a common interface for compression methods and supports distributed training.

Related

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    intel/neural-compressor

    An open-source Python library that provides model compression and quantization techniques for PyTorch, TensorFlow, and JAX models to improve inference efficiency.

  • Project
    pnnx/pnnx

    pnnx is an open‑source tool that optimises PyTorch models and exports them to a lightweight, dependency‑free format (PNNX) and to ncnn/ONNX‑zero files, enabling fast inference on edge devices.

  • Project
    Neroued/ninfer

    A C++/CUDA inference engine optimized for the NVIDIA RTX 5090, providing maximum performance for Qwen3.5 Dense and MoE models across text, image, and video.

  • Project
    openvinotoolkit/model_server

    OpenVINO Model Server (OVMS) is a C++ inference server that serves LLMs, vision‑language models, and classic deep‑learning models via OpenAI‑compatible REST or KServe gRPC APIs. Optimized for Intel CPUs, GPUs and NPUs, it provides features like continuous batching, streaming, model versioning, hot‑reload, and Prometheus metrics. Deployable via Docker, bare‑metal binaries, or Kubernetes, it supports many model formats (TensorFlow, ONNX, OpenVINO IR, GGUF, etc.) and integrates with standard client libraries.