huggingface/optimum-quanto

What it solves

Optimum Quanto is a PyTorch quantization backend designed to reduce the memory footprint and potentially increase the inference speed of AI models. It allows users to convert high-precision floating-point models into lower-precision representations (such as int8, int4, or float8) without requiring complex model tracing, making it easier to deploy models on various devices including CUDA and MPS.

How it works

Quanto operates by replacing standard PyTorch modules (like Linear and Conv2d) with quantized versions. It uses a specialized Tensor subclass to project source tensors into an optimal range for the destination type to minimize accuracy loss. The workflow typically involves quantizing a model (initially in dynamic mode), optionally calibrating it to find the best activation scales, and finally "freezing" the weights to replace floating-point weights with integer weights for permanent storage and faster execution.

Who it’s for

It is primarily intended for developers and researchers working with PyTorch and Hugging Face models (such as LLMs and Diffusers) who need to compress their models to fit on smaller hardware or improve inference efficiency.

Highlights

• Versatile Precision: Supports weights in int2, int4, int8, and float8, and activations in int8 and float8.
• Seamless Integration: Provides helper classes for one-click quantization of Hugging Face CausalLM and Diffusers models.
• Hardware Flexibility: Works in eager mode and supports deployment across CUDA, MPS, and other devices.
• Optimization: Includes accelerated matrix multiplications for CUDA devices across various mixed-precision combinations.
• Calibration and Tuning: Supports a calibration mode for activations and Quantization-Aware-Training (QAT) to recover accuracy loss.