SonySemiconductorSolutions/mct-model-optimization

Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.

What it solves

It provides a toolkit for compressing and optimizing pre-trained floating-point AI models (PyTorch and Keras) to make them efficient enough for deployment on edge hardware, where memory and computational resources are limited.

How it works

The toolkit uses several model compression techniques to reduce the precision and size of models:

  • Quantization: It converts high-precision floating-point weights and activations into lower-precision formats. It supports Post Training Quantization (PTQ), Gradient-based Post Training Quantization (GPTQ), and Quantization Aware Training (QAT).
  • Pruning: It uses structured pruning to remove redundant input channels from layers and reconstruct weights to reduce complexity.
  • Hardware-Awareness: It utilizes Target Platform Capabilities (TPC) to ensure optimizations respect the constraints of the specific target hardware.
  • Data Generation: If real image data is unavailable for quantization, it can generate synthetic images based on the statistics in the model's batch normalization layers.

Who it’s for

AI engineers and developers who need to deploy large PyTorch or Keras models to edge devices and need to reduce their size and computational overhead without sacrificing too much accuracy.

Highlights

  • Mixed-precision search: Automatically assigns the optimal bit-width per layer for weights and activations.
  • XQuant: An explainable quantization tool that provides reports, histograms, and similarity metrics to troubleshoot performance drops.
  • EPTQ: An advanced Hessian-guided network-wise optimization algorithm for enhanced post-training quantization.
  • Broad Framework Support: Compatible with PyTorch (2.3 to 2.6) and TensorFlow/Keras (2.14 to 2.15).