kendryte/nncase

Open deep learning compiler stack for Kendryte AI accelerators ✨

What it solves

nncase is a neural network compiler designed to optimize and deploy AI models on AI accelerators, specifically targeting hardware like the K210, K510, and K230 chips. It bridges the gap between high-level model formats and the efficient execution of these models on specialized hardware.

How it works

The compiler takes models from common formats such as TFLite, Caffe, and ONNX, and compiles them into a format optimized for the target AI accelerator. It employs several optimization techniques:

  • Operator Fusion: Combines multiple operations into a single step to reduce overhead.
  • Memory Management: Uses static memory allocation to avoid heap memory acquisition during runtime.
  • Inference Support: Supports both floating-point and quantized uint8 inference, including post-training quantization using a calibration dataset.
  • Zero-Copy Loading: Uses a flat model format that allows the model to be loaded without copying data.

Who it’s for

Developers and engineers working with AI accelerators (K210, K510, K230) who need to compile and deploy optimized neural networks for image classification, object detection, pose estimation, and image segmentation.

Highlights

  • Broad Format Support: Compatible with TFLite, Caffe, and #{ "off_topic": false, "markdown": "## What it solves

nncase is a neural network compiler designed to optimize and deploy AI models on AI accelerators, specifically accelerators like the K210, K510, and K230 chips. It bridges the gap between high-level model formats and the ONNX formats optimized for the target hardware.

How it works

The compiler processes models from TFLite, Caffe, and ONNX formats. It optimizes the model for the target AI accelerator through:

  • Operator Fusion: Combining multiple operations to improve efficiency.
  • Static Memory Allocation: Ensuring no heap memory is acquired during runtime for predictable performance.
  • Quantization: Supporting both float and quantized uint8 inference, including post-training quantization with calibration datasets.
  • Zero-Copy Loading: Utilizing a flat model format for efficient loading.

Who it’s for

Developers and engineers deploying AI models to specialized AI accelerators (such as the K210, K510, and K230), particularly for computer vision tasks like image classification, object detection, and pose estimation.

Highlights

  • Multi-Format Support: Supports TFLite, Caffe, and ONNX operators.
  • Hardware Optimization: Specifically optimized for K210, K510, and K230 accelerators.
  • Efficient Memory: Uses static memory allocation and zero-copy loading.
  • Versatile Inference: Supports both float and quantized uint8 inference.

Related

  • Project
    PINTO0309/PINTO_model_zoo

    A comprehensive collection of AI models pre-converted and quantized across various frameworks like TensorFlow, PyTorch, ONNX, and CoreML for easier deployment on edge devices.

  • Project
    onnx/models

    A curated collection of pre-trained, state-of-the-art machine learning models in the ONNX format, enabling cross-framework compatibility and deployment.

  • Project
    microsoft/Olive

    An AI model optimization toolkit that composes the best optimization techniques to create efficient ONNX models for cloud or edge inference based on hardware and performance constraints.

  • Project
    NVIDIA/Model-Optimizer

    A library for accelerating AI models through quantization, pruning, and distillation, enabling faster inference and reduced memory usage across various deployment frameworks.

  • Project
    microsoft/onnxruntime

    A cross-platform machine learning accelerator that optimizes inference and training for models from various frameworks like PyTorch and TensorFlow.