Intel AutoRound: Advanced Weight-Only Quantization for LLMs and VLMs
Intel's AutoRound is a weight-only post-training quantization (PTQ) tool designed to reduce the size and inference latency of large language models (LLMs) and vision-language models (VLMs). By employing signed gradient descent to jointly optimize weight rounding and clipping ranges, AutoRound enables accurate low-bit quantization (INT2 through INT8) with minimal accuracy loss.
Superior Accuracy at Low Bit Widths
AutoRound demonstrates significant performance gains in low-bit scenarios, particularly at 2-bit precision. According to the provided data, AutoRound can outperform popular baselines by up to 2.1x in relative accuracy at INT2. It also maintains a competitive edge at 4-bit precision, as evidenced by its performance on the Low-Bit Open LLM Leaderboard.
Technical Capabilities and Flexibility
AutoRound is designed for efficiency and broad compatibility across different model architectures and hardware backends.
Model and Device Support
- LLMs: Supports most popular architectures, including LLaMA, Qwen, and DeepSeek.
- VLMs: Supports over 10 vision-language models, including Gemma3 and Mistral-Small-3.1. For unsupported models, an RTN method can be applied using
--iters 0, though some accuracy loss is expected. - Hardware: Compatible with CPUs, Intel GPUs, and CUDA.
Quantization Configurations and Formats
AutoRound supports a variety of weight-only configurations, including INT2, INT3, INT4, INT8, and mixed-bit tuning. The tool can export models in several formats to ensure interoperability, including:
- AutoRound
- GPTQ
- AWQ
- Select GGUF formats
Quantization Efficiency and Performance
AutoRound achieves high accuracy using only 200 tuning steps and a small calibration dataset of as few as 128 samples. This makes it significantly faster and less resource-intensive than other INT2 methods.
Quantization Speed Benchmarks
In "light mode," AutoRound can quantize a 72B parameter model in 37 minutes on an NVIDIA A100 GPU. The following table compares quantization times across different tools and configurations:
| Model | AutoAWQ (128 samples) | AutoAWQ (512 samples) | GPTQ (Transformers) | AutoRound Light (128 samples) | AutoRound (128 samples) | AutoRound (512 samples) |
|---|---|---|---|---|---|---|
| Qwen2.5 3B | 7min | 17min | 13min | 3min | 8min | 9min |
| Llama3.1-8B | 13min | 27min | 22min | 6min | 13min | 17min |
| Qwen2.5 72B | 105min | 230min | OOM | 37min | 120min | 149min |
Tuning Recipes
AutoRound provides three distinct recipes to balance accuracy and speed:
auto-round-best: Optimized for maximum accuracy, recommended for 2-bit quantization.auto-round(Default): Provides a balanced trade-off between accuracy and tuning cost.auto-round-light: Optimized for maximum speed.
Implementation and Usage
AutoRound can be installed via pip (pip install auto-round) and supports both command-line and API-based workflows.
Command Line Interface
Users can specify the model, bit-width, group size, and desired export formats. For example:
auto-round --model Qwen/Qwen3-0.6B --bits 4 --group_size 128 --format "auto_round,auto_awq,auto_gptq" --output_dir ./tmp_autoround
API Integration
AutoRound integrates with the transformers library, allowing users to load a model and tokenizer, initialize the AutoRound class, and save the quantized model using quantize_and_save. It also supports converting existing GPTQ or AWQ models into the AutoRound format to improve compatibility with Intel devices.