intel/neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

What it solves

Intel Neural Compressor addresses the challenge of deploying large deep learning models by reducing their size and computational requirements. It allows developers to compress models without sacrificing significant performance, making them more efficient to run on various hardware platforms.

How it works

The library provides a suite of model compression techniques that can be applied to models built with PyTorch, TensorFlow, and JAX. It implements several quantization methods, including:

  • Static and Dynamic Quantization: Reducing the precision of weights and activations.
  • SmoothQuant: A technique to optimize quantization for specific model architectures.
  • Weight-Only Quantization: Compressing only the model weights to save memory.
  • Quantization-Aware Training (QAT): Integrating quantization into the training process.
  • Mixed Precision: Using different precision levels for different parts of the model.
  • Advanced LLM/VLM Quantization: Specialized support for Large Language Models and Vision-Language Models via integration with AutoRound.

Who it’s for

It is designed for AI engineers and researchers who need to optimize deep learning models for deployment on Intel hardware (such as Gaudi AI Accelerators, Xeon processors, and Data Center GPUs), as well as other platforms like AMD, ARM, and NVIDIA GPUs.

Highlights

  • Multi-Framework Support: Works across PyTorch, TensorFlow, and JAX.
  • Broad Hardware Compatibility: Optimized for Intel Gaudi, Core Ultra, Xeon, and Data Center GPUs, with limited support for other CPUs and GPUs.
  • LLM/VLM Specialization: Dedicated tools for compressing models like LLaMA, Qwen, and DeepSeek.
  • Cutting-Edge Precision: Support for experimental low-precision formats including FP8, MXFP8, MXFP4, and NVFP4.

Related

  • Project
    SonySemiconductorSolutions/mct-model-optimization

    A model compression toolkit for PyTorch and Keras that optimizes AI models for edge deployment using quantization, pruning, and hardware-aware optimization.

  • Project
    vllm-project/llm-compressor

    A library for quantizing and compressing large language models (weights, activations, KV‑cache, etc.) so they run efficiently with the vLLM inference engine. Supports many algorithms (GPTQ, AWQ, AutoRound, REAP pruning, etc.) and formats (FP8, NVFP4, INT4, mixed‑precision). Provides one‑line API, DDP + disk‑offloading for multi‑terabyte models, and pre‑quantized checkpoints for popular LLMs.

  • Project
    pytorch/ao

    A PyTorch-native library for model optimization that provides quantization and sparsity tools to accelerate training and inference while reducing memory usage.

  • Project
    Xilinx/brevitas

    A PyTorch library for neural network quantization that supports both post-training quantization and quantization-aware training to improve model efficiency.

  • Project
    vllm-project/compressed-tensors

    A library that extends the safetensors format to provide a unified, extensible way to store and manage compressed tensor data for LLMs across various quantization and sparsity schemes.