sophgo/tpu-mlir

What it solves

TPU-MLIR is a machine-learning compiler that allows developers to take pre-trained neural networks from popular frameworks (like PyTorch, ONNX, TFLite, and Caffe) and convert them into a specialized format called bmodel files. These files are optimized to run efficiently on TPU hardware, bridging the gap between high-level AI model design and hardware-specific execution.

How it works

Built on the MLIR (Multi-Level Intermediate Representation) framework, the compiler uses a pipeline of "dialects" and pattern rewrites to lower high-level model descriptions into hardware-specific instructions. It includes a comprehensive toolchain for quantization (converting models from F32 to BF16, F16, or INT8) and calibration to maintain accuracy. For Large Language Models (LLMs), it provides a dedicated llm_convert.py tool for one-shot conversion of HuggingFace models, supporting advanced features like KV-cache management and chunked inference.

Who it’s for

This tool is designed for AI engineers and hardware developers who need to deploy optimized AI models—ranging from vision models like YOLOv5 to LLMs like Qwen—onto TPU-based accelerators.

Highlights

• Multi-framework support: Directly converts models from PyTorch, ONNX, TFLite, and Caffe.
• LLM Optimization: Specialized support for HuggingFace LLMs with one-shot conversion and KV-cache handling.
• Comprehensive Quantization: Supports F32, BF16, F16, and INT8 (symmetric/asymmetric), as well as AWQ, GPTQ, and AutoRound.
• Production Tooling: Includes a simulator, visualizer, and tools for accuracy validation and bmodel inspection.
• MLIR-based Pipeline: Uses a clean lowering pipeline with layer-group memory planning for efficient hardware utilization.