intel/auto-round

A SOTA quantization toolkit for high-accuracy low-bit LLM inference|简洁且高效的量化工具包

intel/auto‑round – Advanced Low‑Bit Quantization Toolkit for LLMs & VLMs

What it is – AutoRound is a Python library (Python 3.10+) that quantizes large language models (LLMs) and vision‑language models (VLMs) to ultra‑low bit‑widths (2‑4 bits) while preserving most of the original accuracy. It does this with a “sign‑gradient descent” rounding algorithm and a collection of optional post‑processing steps (AWQ, Hadamard, etc.).

Why it matters – Quantization reduces model size and inference latency, enabling deployment on CPUs, Intel GPUs (XPU), NVIDIA GPUs, Habana Gaudi HPUs, and even on‑device environments. AutoRound claims state‑of‑the‑art accuracy at 2‑3 bits and seamless integration with popular inference stacks such as Transformers, vLLM, SGLang, and LLM‑Compressor.

Key Features (as listed in the README)

High accuracy at 2–4 bits – benchmarked on dozens of models, with examples showing >97 % of original performance for INT2‑mixed models.
Broad hardware support – works on CPU (Xeon), Intel XPU, CUDA GPUs, and Habana Gaudi HPUs.
Multiple export formats – can output models in AutoRound’s native format, AutoAWQ, AutoGPTQ, and GGUF (including many GGUF quantization types).
Fast mixed‑precision scheme generation – the AutoScheme utility creates per‑layer mixed‑bit recipes in minutes, using only ~1.1‑1.5× the model’s BF16 RAM as overhead.
Various quantization recipes – auto-round, auto-round-best, auto-round-light, auto-round-opt-rtn, auto-round-rtn etc., letting users trade speed vs. accuracy.
Torch‑compile acceleration – optional torch.compile speed‑up (enabled by default on non‑Windows platforms).
Calibration‑free / model‑free modes – can run without a calibration dataset for certain schemes, and a free‑device service is offered for such runs.
Support for 10+ vision‑language models – out‑of‑the‑box pipelines for multimodal models.
Experimental extensions – algorithm composition (--algs awq,signround), block‑wise FP8, MXFP/NVFP datatypes, and mixed‑precision AutoScheme.

Installation

# CPU or CUDA GPU
pip install auto-round            # stable release
pip install auto-round-nightly    # nightly build

# Habana Gaudi (HPU) – install inside the Habana container
pip install auto-round-hpu

# Intel XPU (GPU)
pip install torch --index-url https://download.pytorch.org/whl/xpu
pip install auto-round

*Source builds are also supported via pip install . or python setup.py install hpu for Gaudi.

Quick Start (CLI)

auto-round \
  --model Qwen/Qwen3-0.6B \
  --scheme "W4A16" \
  --format "auto_round" \
  --output_dir ./quantized_qwen

Replace --scheme with any supported scheme (e.g., W2A16, MXFP4, GGUF:Q4_K_M).

Quick Start (Python API)

from auto_round import AutoRound

ar = AutoRound(
    model_name_or_path="Qwen/Qwen3-0.6B",
    scheme="W4A16",
    iters=200,               # number of tuning iterations (0 = pure RTN)
    low_gpu_mem_usage=False # set True to save VRAM at the cost of speed
)

ar.quantize_and_save(output_dir="./qmodel", format="auto_round")

The API mirrors the CLI options; key arguments include scheme, bits, group_size, iters, lr, dataset, device_map, and the experimental enable_alg_ext flag for algorithm extensions.

Ecosystem Integration

Transformers – quantized models can be loaded directly with AutoModelForCausalLM.from_pretrained(..., device_map="auto") after conversion.
vLLM – native support (including the experimental AutoScheme WOQ deployment) via the vLLM‑Omni branch.
SGLang – documented quantization hooks.
LLM‑Compressor – a plug‑in that forwards AutoRound’s recipes.
GGUF – export to the GGUF format for use with llama.cpp and other GGUF‑aware runtimes.

Documentation & Resources

User Guide – step‑by‑step tutorial: docs/step_by_step.md (English & Chinese).
Algorithm composition – docs/algorithm_combinations.md.
AutoScheme accuracy – docs/auto_scheme_acc.md.
Benchmark leaderboard – Hugging Face space Intel/low_bit_open_llm_leaderboard.
Research papers – SignRoundV1 (arXiv 2309.05516) and SignRoundV2 (arXiv 2512.04746).

Who might use this?

ML engineers looking to shrink LLMs for edge or cost‑constrained inference.
Researchers experimenting with ultra‑low‑bit quantization algorithms.
Product teams needing a drop‑in quantizer that works with existing Hugging Face pipelines and major inference servers.

TL;DR

AutoRound is Intel’s open‑source, pip‑installable toolkit that lets you quantize LLMs/VLMs to 2‑4 bit representations with minimal accuracy loss, supports a wide hardware spectrum, and plugs into the major model serving stacks.