microsoft/VPTQ

VPTQ, A Flexible and Extreme low-bit quantization algorithm

What it solves

VPTQ addresses the challenge of compressing Large Language Models (LLMs) to extremely low bit-widths (under 2 bits) without requiring retraining, while maintaining high accuracy and minimizing inference overhead.

How it works

It utilizes a novel Post-Training Quantization (PTQ) method based on Vector Quantization. This approach allows the compression of massive models (such as Llama-3.1 405B) to 1-2 bits. The process is designed to be lightweight, taking approximately 17 hours to quantize a 405B parameter model.

Who it’s for

AI developers and researchers who need to deploy massive LLMs on hardware with limited memory (e.g., running a 70B model on a single RTX 4090) and those looking for high-throughput inference with low decode overhead.

Highlights

  • Extreme Compression: Supports compression to less than 2 bits for models up to 405B parameters.
  • High Efficiency: Lightweight quantization algorithm and agile inference with optimized Time To First Token (TTFT) and throughput.
  • Broad Integration: Integrated into the Hugging Face Transformers library.
  • Wide Model Support: Compatible with various model families including Llama 3.1/3.3, DeepSeek R1, Mistral, and Qwen.

Related

  • Project
    ModelCloud/GPTQModel

    A comprehensive LLM quantization toolkit that compresses models using methods like GPTQ and AWQ with hardware acceleration for NVIDIA, AMD, Intel, and Huawei NPUs.

  • Project
    vllm-project/llm-compressor

    LLM Compressor is a Python library that quantizes and prunes large language models into the `compressed‑tensors` format, enabling memory‑efficient deployment with vLLM. It supports many low‑precision formats (NVFP4, FP8, INT4, etc.), several PTQ/GPTQ algorithms, DDP and disk‑offloading for huge models, and ships pre‑quantized checkpoints for popular LLMs.

  • Dispatch
    Hugging Face AutoGPTQ and Transformers Integration

    Hugging Face has integrated the AutoGPTQ library into Transformers, enabling the quantization of LLMs to 8, 4, 3, or 2-bit precision to reduce memory requirements with negligible accuracy loss at 4-bit.

  • Project
    intel/auto-round

    AutoRound is Intel’s open‑source Python toolkit for ultra‑low‑bit (2‑4 bit) quantization of large language and vision‑language models. It provides fast, high‑accuracy recipes, supports CPUs, Intel GPUs, NVIDIA CUDA, and Habana Gaudi, and integrates with Transformers, vLLM, and SGLang. Install via pip, run a one‑line CLI or use the Python API, then deploy the quantized model with any supported backend.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.