pytorch/ao

PyTorch native quantization for training and inference

What it solves

TorchAO provides a native PyTorch library for optimizing models from training to serving. It addresses the challenge of reducing model size and increasing inference speed without sacrificing significant accuracy, specifically by providing tools for quantization and sparsity.

How it works

TorchAO integrates directly with torch.compile() and FSDP2, allowing it to work out-of-the-box with most HuggingFace models. It implements various low-precision data types (dtypes) and memory layouts, using kernels in PyTorch, CUDA, C++, or Triton to accelerate performance on GPUs and CPUs.

Who it’s for

It is designed for machine learning engineers and researchers who need to deploy large models (like LLMs or diffusion models) more efficiently on server GPUs, edge devices via ExecuTorch, or ARM CPUs.

Highlights

  • Quantization-Aware Training (QAT): Recovers accuracy degradation caused by post-training quantization, particularly for int4 models.
  • Float8 Training: Accelerates pre-training (e.g., Llama-3.1-70B) by up to 1.5x using scaled float8 dtypes.
  • Inference Optimization: Supports int4 weight-only and float8 dynamic quantization for significant speedups and memory reduction.
  • Memory-Efficient Optimizers: Includes quantized optimizers (AdamW 4-bit/8-bit/Fp8) and CPU offloading to reduce VRAM requirements.
  • C-level Integrations: Integrated with vLLM, HuggingFace Transformers/Diffusers/PEFT, Unsloth, and SGLang.

Related

  • Project
    bitsandbytes-foundation/bitsandbytes

    A PyTorch library that enables accessible large language models through 4-bit and 8-bit quantization for both inference and training.

  • Project
    huggingface/optimum-quanto

    A PyTorch quantization backend for Optimum that reduces model size and memory usage by converting weights and activations to lower precision like int4 or float8.

  • Project
    Xilinx/brevitas

    A PyTorch library for neural network quantization that supports both post-training quantization and quantization-aware training to improve model efficiency.

  • Project
    pypose/bae

    bae is a PyTorch library for exact, GPU-accelerated bundle adjustment and pose graph optimization, built on PyPose and designed for robotics and vision applications.

  • Project
    intel/neural-compressor

    An open-source Python library that provides model compression and quantization techniques for PyTorch, TensorFlow, and JAX models to improve inference efficiency.