deepseek-ai/TileKernels

What it solves

TileKernels provides a library of highly optimized compute kernels for common operations used in Large Language Model (LLM) training and inference. It aims to achieve performance near the hardware's theoretical limits of compute or memory bandwidth, reducing bottlenecks in complex AI workloads.

How it works

The library is implemented using TileLang, a domain-specific language (DSL) that supports multiple hardware backends. This allows the same Python APIs to run efficiently on both NVIDIA GPUs (SM90/SM100) and Huawei Ascend NPUs (950), with the backend selected automatically at runtime.

Who it’s for

It is designed for developers and researchers working on high-performance LLM training and inference, particularly those using Mixture-of-Experts (MoE) or advanced quantization techniques.

Highlights

• MoE Routing: Optimized top-k expert selection and scoring.
• Quantization: Supports per-token, per-block, and per-channel FP8/FP4 casting and dequantization, including fused SwiGLU operations.
• Engram: Gating kernels with fused RMSNorm and support for forward/backward passes.
• Manifold HyperConnection: Includes Sinkhorn normalization and mix splitting/application kernels.
• Hardware Versatility: Native support for both NVIDIA and Huawei Ascend hardware backends.