Xilinx/brevitas

Brevitas: neural network quantization in PyTorch

What it solves

Brevitas is designed to make neural network quantization easier and more flexible. It allows developers to reduce the precision of model weights and activations to lower the memory footprint and increase inference speed without significantly sacrificing accuracy.

How it works

It is a PyTorch library that provides quantized versions of standard deep neural network layers, such as convolutional layers (QuantConv2d), recurrent layers (QuantLSTM), and attention mechanisms (QuantMultiheadAttention). The library supports two primary methods of quantization:

  • Post-Training Quantization (PTQ): Quantizing a model after it has already been trained.
  • Quantization-Aware Training (QAT): Integrating quantization into the training process itself to help the model adapt to lower precision.

Users can individually tune the quantization settings for different tensors, including inputs, weights, bias, and outputs, adjusting parameters like bit-width and scale granularity.

Who it’s for

This tool is for AI researchers and developers who need to optimize neural networks for deployment on hardware that benefits from low-precision arithmetic, such as FPGAs or other specialized accelerators.

Highlights

  • Support for both PTQ and QAT.
  • Quantized implementations of common PyTorch layers (Conv1d, Conv2d, RNN, LSTM, MultiheadAttention).
  • Highly tunable quantization settings for weights, inputs, and biases.
  • Integration with the PyTorch ecosystem.

Related

  • Project
    bitsandbytes-foundation/bitsandbytes

    A PyTorch library that enables accessible large language models through 4-bit and 8-bit quantization for both inference and training.

  • Project
    pytorch/ao

    A PyTorch-native library for model optimization that provides quantization and sparsity tools to accelerate training and inference while reducing memory usage.

  • Project
    huggingface/optimum-quanto

    A PyTorch quantization backend for Optimum that reduces model size and memory usage by converting weights and activations to lower precision like int4 or float8.

  • Project
    torch-uncertainty/torch-uncertainty

    A deep learning framework for uncertainty quantification that provides tools to make neural networks more reliable through uncertainty-aware training, evaluation, and post-processing.