intel/auto-round

AutoRound – Post‑Training Quantization for LLMs & VLMs

What it is – AutoRound is a Python library that lets you shrink large language models (LLMs) and vision‑language models (VLMs) to very low‑bit formats (2‑4 bits for weights, plus a few mixed‑precision options) after the model has already been trained. The goal is to keep the model’s accuracy close to the original (often within a few percent) while dramatically reducing memory‑footprint and inference latency.

Why it matters – Running LLMs on‑device or on cheaper hardware (CPU, Intel XPU, Gaudi HPUs, or even consumer‑grade GPUs) normally requires a lot of RAM/VRAM. Quantization cuts the size of the weight tensors, enabling:

• Faster inference on the same hardware.
• Deployment on cheaper or edge devices.
• Lower cost for serving many concurrent requests.

Key capabilities

• Multiple low‑bit schemes – supports 2‑bit, 3‑bit, 4‑bit, 5‑bit, 6‑bit, 7‑bit weight‑only quantization (WOQ) as well as mixed‑precision recipes (AutoScheme). Specialized dtypes like MXFP4, NVFP4, FP8, and the GGUF family are also available.
• Algorithms – built‑in implementations of SignRoundV1/V2, AWQ, GPTQ, and a new NeUQI grid‑search for calibration‑free quantization. Users can chain algorithms (e.g., awq,signround).
• Ecosystem integration – works directly with 🤗 Transformers, vLLM, and SGLang. Quantized models can be exported to formats used by those runtimes (auto_round, auto_awq, auto_gptq, GGUF, etc.).
• Hardware support – runs on Intel CPUs/XPUs, NVIDIA GPUs (CUDA), and Habana Gaudi HPUs. The package provides separate wheels for each platform.
• Fast, affordable tuning – a 7 B model can be quantized in ~10 minutes on a single GPU; the library offers “light”, “best”, and “opt‑rtn” preset recipes that trade speed for accuracy.
• Vision‑language support – out‑of‑the‑box quantization for more than ten VLMs, with experimental full‑model quantization beyond the text encoder.

How you use it

1. Install – pip install auto-round (or the nightly / HPU‑specific wheels).
2. CLI – run auto-round --model <repo>/<model> --scheme "W4A16" --format auto_round --output_dir ./qmodel.
   • Choose a preset recipe (auto-round-best, auto-round-light, auto-round-opt-rtn, auto-round-rtn).
3. Python API –

   from auto_round import AutoRound
   ar = AutoRound("Qwen/Qwen3-0.6B", scheme="W4A16", iters=200)
   ar.quantize_and_save("./qmodel", format="auto_round")
   

   • Adjust bits, group_size, iters, lr, dataset, etc., to control accuracy vs. speed.
4. Run inference – load the quantized checkpoint with vLLM, Transformers, or any runtime that supports the exported format.

Typical workflow

• Pick a model and a target bit‑width (e.g., 4‑bit weight‑only).
• Run AutoRound with the default auto-round recipe for a quick result, or auto-round-best for maximum accuracy (more tuning iterations).
• If you need mixed‑precision, enable AutoScheme (--enable_alg_ext) to let the tool automatically search a per‑layer bit‑width plan.
• Export to the format required by your serving stack (e.g., GGUF for llama.cpp, auto_awq for AWQ‑based runtimes).

Who benefits

• Researchers who want to experiment with ultra‑low‑bit LLMs without writing custom quantization code.
• Engineers deploying LLMs on cost‑constrained servers or edge devices.
• Anyone needing a drop‑in quantization step that integrates with the popular 🤗 Transformers/vLLM ecosystem.

---

Resources

• Docs & tutorials – step‑by‑step guides in docs/step_by_step.md and Chinese version step_by_step_CN.md.
• Benchmarks – accuracy tables for 2‑3‑bit models, GGUF leaderboard, and MoE speed‑up notes.
• Paper links – SignRoundV1 (arXiv 2309.05516) and SignRoundV2 (arXiv 2512.04746).
• Model zoo – pre‑quantized checkpoints hosted on HuggingFace under the Intel organization.

Bottom line – AutoRound is a production‑ready, open‑source toolkit that makes post‑training quantization of large language and vision‑language models practical across a wide range of hardware, while preserving most of the original model’s performance.