thu-pacman/chitu

What it solves

Chitu is a production-grade large model inference engine designed to bridge the gap between small-scale AI experiments and large-scale enterprise deployments. It addresses the need for a high-performance inference framework that is flexible enough to handle diverse hardware environments and stable enough to support concurrent business traffic.

How it works

Chitu provides a scalable inference solution that supports a wide range of hardware configurations, from pure CPU deployments and single GPU setups to large-scale clusters. It utilizes efficient operators for online quantization conversion (such as FP4 to FP8/BF16) and supports heterogeneous mixed inference (CPU+GPU) to enable the deployment of massive models like DeepSeek-R1 671B on limited hardware.

Who it’s for

It is intended for enterprises and developers who need to deploy large language models (LLMs) in production environments across various hardware backends, including NVIDIA GPUs and various domestic Chinese AI chips (e.g., Ascend, Moore Threads, Muxi, Haiguang).

Highlights

• Broad Hardware Compatibility: Supports NVIDIA GPUs, Ascend 910B, Moore Threads, Muxi, and Haiguang chips.
• Flexible Deployment: Offers a single-file executable (chitu.run) for complex tasks like multi-node, multi-instance, and PD separation.
• High-Performance Quantization: Implements efficient operators for FP4, FP8, and BF16 conversions.
• Wide Model Support: Compatible with major models including DeepSeek, Qwen, GLM, and Kimi.