aimet: a toolkit for quantizing and compressing ML models to reduce memory footprint and increase inference speed on edge devices

What it solves

AIMET reduces the compute load and memory footprint of trained machine learning models, making them faster and more efficient for deployment on edge devices like mobile phones and laptops. It specifically addresses the challenge of maintaining model accuracy when converting high-precision (32-bit floating-point) models to lower-precision (8-bit or 4-bit integer) formats.

How it works

AIMET provides a suite of tools for quantization and compression. It supports PyTorch and ONNX models using the following methods:

• Post-Training Quantization (PTQ): Uses techniques like Calibration, AdaRound (Adaptive Rounding), SeqMSE, and SpinQuant to optimize weights and activations without requiring full retraining.
• Quantization Aware Training (QAT): Integrates quantization simulation into the training process to further minimize accuracy loss.
• Model Compression: Employs Spatial SVD (tensor decomposition) and Channel Pruning to remove redundant parameters and reduce multiply-accumulate (MAC) operations.
• Data-Free Quantization (DFQ): Allows for quantization without the need for original training data.

Who it’s for

ML engineers and developers who need to optimize deep learning models for high-performance inference on edge hardware, particularly those using PyTorch or ONNX frameworks.

Highlights

• Significant Speedups: Enables models to run up to 15x faster on specific hardware like the Qualcomm Hexagon DSP compared to CPUs.
• # 4x Memory Reduction: 8-bit precision models occupy significantly less space than 32-bit models.
• Automated Optimization: Provides APIs to automate the optimization process, reducing the need for manual tweaking.
• Broad Support: Works with various model types, including CNNs (ResNet, MobileNet) and recurrent models (RNN, LSTM, GRU).