intel/auto-round

A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.

What it solves

AutoRound is a quantization toolkit that allows Large Language Models (LLMs) and Vision-Language Models (VLMs) to be compressed to ultra-low bit widths (2–4 bits) while maintaining high accuracy. It reduces the memory footprint and computational requirements of these models, making them easier to deploy on various hardware without significant performance loss.

How it works

The toolkit uses a sign-gradient descent algorithm to optimize the rounding process during quantization. It offers several "recipes" (such as auto-round-best for maximum accuracy or auto-round-rtn for the fastest baseline) to balance speed and precision. It also features an AutoScheme API that automatically generates mixed-precision quantization recipes to optimize the trade-off between model size and accuracy.

Who it’s for

It is designed for AI developers and researchers who need to deploy LLMs or VLMs on hardware with limited memory, as well as those integrating quantized models into production environments using frameworks like vLLM, SGLang, or Transformers.

Highlights

  • High Accuracy at Low Bits: Maintains strong performance even at 2–3 bits.
  • Broad Ecosystem Integration: Works seamlessly with Transformers, vLLM, SGLang, and LLM-Compressor.
  • Multiple Export Formats: Supports AutoRound, AutoAWQ, AutoGPTQ, and GGUF formats.
  • Efficient Tuning: Can quantize a 7B model in approximately 10 minutes on a single GPU.
  • VLM Support: Out-of-the-box quantization for over 10 vision-language models.
  • Hardware Compatibility: Supports CPU (Xeon), GPU (CUDA, Intel GPU), and HPU (Gaudi).