ARM-software/CMSIS-NN

CMSIS-NN Library

What it solves

CMSIS-NN provides a collection of highly efficient neural network kernels designed to maximize performance and minimize memory usage for AI models running on Arm Cortex-M processors. It solves the challenge of deploying neural networks on resource-constrained microcontrollers by providing optimized implementations of common operators.

How it works

The library implements neural network operators using three levels of optimization based on the target hardware's capabilities:

  • Pure C: Reference implementations that work on all processors (e.g., Cortex-M0, M3).
  • DSP Extension: Uses SIMD instructions for processors like Cortex-M4 or M33.
  • MVE Extension: Uses Arm Helium Technology (M-profile Vector Extension) for high-performance vector processing on cores like Cortex-M55 or M85.

It primarily follows the int8 and int16 quantization specifications of TensorFlow Lite for Microcontrollers (TFLM) to ensure bit-exact compatibility, though it also offers experimental float16 and float32 APIs for MVE-enabled CPUs.

Who it’s for

It is intended for developers deploying machine learning models to Arm Cortex-M microcontrollers, particularly those using frameworks like TensorFlow Lite for Microcontrollers or ExecuTorch.

Highlights

  • Hardware-Specific Optimization: Tailored kernels for Pure C, DSP, and MVE (Helium) architectures.
  • Broad Operator Support: Includes Conv2D, DepthwiseConv2D, Fully Connected, LSTM, and various pooling and activation functions.
  • Quantization Support: Bit-exact compatibility with TFLM int8 and int16 specifications.
  • Experimental Float Support: Provides float16 and float32 APIs for specific use cases where quantization is not feasible.
  • Python Bindings: Includes pybind11-based helpers to calculate host buffer sizes.

Related

  • Project
    deepseek-ai/DeepGEMM

    A high-performance CUDA kernel library for LLMs that provides optimized GEMM and fused MoE primitives for NVIDIA SM90 and SM100 GPUs.

  • Project
    Arm-Examples/ML-examples

    A collection of machine learning examples and tutorials for deploying and optimizing AI models on Arm Ethos NPUs and Cortex-based platforms.

  • Project
    ARM-software/ComputeLibrary

    A collection of low-level machine learning functions optimized for Arm Cortex-A, Neoverse, and Mali GPU architectures to provide high-performance ML inference.

  • Project
    huggingface/optimum

    An extension of the Hugging Face ecosystem that provides optimization tools to train and run AI models with maximum efficiency on targeted hardware accelerators.

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a Python library that supplies ultra‑fast, hardware‑aware CUDA kernels for LLM inference—covering attention, GEMM, MoE, sampling, and related ops. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM), supports low‑precision FP8/FP4, works with all modern NVIDIA GPUs, and is already used in major serving frameworks like vLLM and TensorRT‑LLM.