pytorch/ao

What it solves

Large AI models are slow and memory-hungry. TorchAO reduces both by quantizing model weights and activations to lower-precision formats (like int4, int8, float8), so models run faster and use less memory during training and inference. It also tackles the accuracy drop that quantization can cause through quantization-aware training (QAT), and it works directly with PyTorch's compile and FSDP2 systems.

How it works

TorchAO provides simple Python APIs that replace a model's weights with quantized versions. For inference, you can apply an Int4WeightOnlyConfig or Float8DynamicActivationFloat8WeightConfig to a model in one line. For training, it offers float8 training recipes that speed up large-scale pretraining, plus QAT workflows that simulate quantization during fine-tuning and then convert the model back to a quantized form. It also includes memory-efficient optimizers (like 8-bit Adam) and CPU offloading to cut GPU memory usage.

Who it's for

Developers and researchers using PyTorch who want to deploy large language models or diffusion models faster, train big models more cheaply, or run them on edge devices like phones. It's ideal for those working with HuggingFace transformers, diffusers, vLLM, or torchtitan, since it integrates directly with these tools.

Highlights

• Up to 1.5-2x training speedups with float8 and 1.89x faster int4 inference with 58% less memory.
• Quantization-aware training recovers up to 96% of the accuracy lost by post-training quantization.
• Works out-of-the-box with torch.compile(), FSDP2, and major HuggingFace libraries.
• Supports server GPUs, CPUs, and edge deployment through ExecuTorch, with pre-quantized models available on HuggingFace Hub.