alibaba/MNN

MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.

What it solves

Deep learning models are often too large and computationally expensive to run on mobile phones, embedded devices, or IoT hardware. MNN provides a highly efficient, lightweight solution to deploy these models locally on various platforms without requiring heavy dependencies.

How it works

MNN uses optimized assembly code for ARM and x64 CPUs and supports hardware acceleration via Metal, OpenCL, Vulkan, and CUDA for GPUs. It includes a converter to transform models from formats like TensorFlow, ONNX, and PyTorch into MNN format, and offers quantization (FP16/Int8) to significantly reduce model size and increase speed.

Who it’s for

  • Mobile and IoT Developers: Who need to run AI models (like LLMs or image editors) locally on Android, iOS, or embedded devices.
  • ML Engineers: Who want to deploy complex models to edge devices using a simple Python API or C++.
  • App Developers: Looking for lightweight computer vision or language model capabilities within existing applications.

Highlights

  • Extremely Lightweight: Core Android library is only ~800KB; iOS static library is ~12MB.
  • Broad Model Support: Compatible with TensorFlow, Caffe, ONNX, and Torchscripts.
  • High Performance: Optimized for various architectures including ARM, x86, and NVIDIA GPUs.
  • Versatile Runtime: Includes specialized solutions for Large Language Models (MNN-LLM) and Stable Diffusion (MNN-Diffusion).