SonySemiconductorSolutions/mct-model-optimization

Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.

What it solves

MCT (Model Compression Toolkit) helps developers shrink trained AI models so they run faster and use less memory on edge devices like phones, cameras, and embedded hardware. It tackles the problem of deploying large neural networks on devices with limited compute and storage by converting them into smaller, quantized versions without destroying accuracy.

How it works

You start with an already-trained floating-point model (in PyTorch or Keras). MCT provides three main quantization flows, each with a different trade-off between complexity and quality:

  • Post-Training Quantization (PTQ): low complexity, takes roughly 1-10 CPU minutes, does not require retraining.
  • Gradient-based Post-Training Quantization (GPTQ): moderate complexity, uses gradients to fine-tune parameters over 1-3 GPU hours.
  • Quantization-Aware Training (QAT): high complexity, retrains the model over 12-36 GPU hours to preserve accuracy.

MCT also includes a mixed-precision search that picks the best bit-width for each layer, graph optimizations, quantization threshold tuning (MSE, no-clipping, MAE), and advanced algorithms like shift-negative correction, outlier filtering, and clustering. It is hardware-aware: the TPC (Target Platform Capabilities) feature lets you describe the constraints of your specific hardware. If you lack a representative dataset, MCT can generate synthetic images from the statistics stored in batch-normalization layers. For model size reduction, it offers structured pruning to remove redundant channels. Debugging and visualization are covered through a network editor, TensorBoard reports, and the XQuant explainable-quantization tool that shows per-layer errors and suggests fixes.

Who it's for

Engineers and researchers working on edge AI applications who need to deploy trained neural networks on resource-constrained devices. It's also useful for anyone who wants to reduce model size or latency while keeping accuracy high, and who works in PyTorch or Keras environments.

Highlights

  • Supports three quantization workflows: PTQ, GPTQ, and QAT.
  • Hardware-aware optimization via Target Platform Capabilities (TPC).
  • Mixed-precision search assigns optimal bit-width per layer.
  • Data-free quantization using synthetic data generation from batch-norm statistics.
  • Structured pruning removes redundant channels to shrink model size.
  • Debugging tools: network editor, TensorBoard visualization, and XQuant explainable reports.

Related

  • Project
    tensorflow/model-optimization

    A suite of tools for optimizing machine learning models for deployment and execution using techniques like quantization and pruning.

  • Project
    huggingface/optimum-quanto

    A PyTorch quantization backend for Optimum that reduces model size and memory usage by converting weights and activations to lower precision like int4 or float8.

  • Project
    Xilinx/brevitas

    A PyTorch library for neural network quantization that supports both post-training quantization and quantization-aware training to optimize models for efficiency.

  • Project
    qualcomm/aimet

    AIMET is Qualcomm’s open‑source toolkit for quantizing and compressing PyTorch/ONNX models, enabling fast, low‑memory inference on edge devices with minimal accuracy loss.

  • Project
    PrunaAI/pruna

    A model optimization framework that uses compression algorithms like quantization and pruning to make AI models faster, smaller, and cheaper to run.