Quanto: a PyTorch quantization backend for Optimum

Hugging Face has released Quanto, a PyTorch quantization backend for the Optimum library. Quanto provides a simplified, versatile approach to quantization, allowing developers to reduce memory storage and computational costs by representing weights and activations in low-precision data types such as 8-bit integers (int8) instead of 32-bit floating point (float32).

Key Features and Capabilities

Quanto is designed for simplicity and versatility, moving beyond the LLM-specific focus of many recent quantization libraries to provide primitives for linear and per-group quantization adaptable to any model modality.

Its core technical capabilities include:

  • Device Agnosticism: Quantized models can be deployed on any device, including CUDA, CPU, and MPS (Apple Silicon).
  • Eager Mode Support: All features are available in eager mode, ensuring compatibility with non-traceable models.
  • Broad Precision Support: It supports weights in int2, int4, int8, and float8, and activations in int8 and float8.
  • Automated Integration: The backend automatically inserts quantization/dequantization stubs, quantized functional operations, and quantized modules.
  • Performance Optimizations: It provides accelerated matrix multiplications on CUDA devices for several combinations, including int8-int8, fp16-int4, bf16-int8, and bf16-int4.
  • Serialization: Compatibility with PyTorch weight_only and Hugging Face Safetensors for efficient model saving and loading.

Quantization Workflow

Quanto implements a structured workflow to move a model from floating-point precision to a frozen quantized state:

  1. Quantize: Converts a standard float model into a dynamically quantized model.
  2. Calibrate (Optional): If activations are quantized, a calibration mode records activation ranges using representative samples.
  3. Tune (Optional): Supports Quantization-Aware-Training (QAT) to recover performance loss through a few epochs of training.
  4. Freeze: Replaces float weights with quantized weights.
  5. Serialize: Saves the quantized weights to a state_dict and the quantization map to a JSON file.
  6. Reload: Uses the requantize helper to instantiate an empty model and reload the serialized weights and map.

Integration with Hugging Face Transformers

Quanto is integrated directly into the transformers library. Users can quantize a model by passing a QuantoConfig to the from_pretrained method.

Technical constraints and optimizations for this integration include:

  • Hardware Requirements: While device-agnostic, float8 requires compatible hardware; otherwise, Quanto silently upcasts to float32 or float16 during matrix multiplication. float8 currently raises errors on MPS devices.
  • Compilation: Quanto is torch.compile friendly. However, for faster generation, users should keep activations=None in QuantoConfig to avoid issues with dynamic quantization (such as QAT or quantized activations).
  • Cross-Modality Use: The integration supports various modalities, demonstrated by the ability to quantize models like openai/whisper-large-v3 in int8.

Performance Benchmarks

Evaluation of meta-llama/Meta-Llama-3.1-8B shows that Quanto provides a path to reduced latency and memory usage. Per-token latency was measured on an NVIDIA A10 GPU. The provided results were obtained without the application of Post-Training-Optimization algorithms such as AWQ or HQQ.

Sources