ROCm/aiter

What it solves

AITER (AI Tensor Engine for ROCm) provides a high-performance library of optimized GPU kernels specifically for AMD GPUs. It solves the problem of framework developers needing production-ready, highly tuned operators for AI inference and training workloads without having to write low-level GPU code themselves.

How it works

AITER acts as a unified collection of operators that can be integrated into various AI frameworks. It utilizes multiple kernel backends to achieve maximum performance, including Triton, Composable Kernel (CK), hand-tuned ASM, and FlyDSL. It offers both C++ and Python APIs, making it accessible for different development levels.

Who it’s for

This library is primarily for AI framework developers (such as those working on vLLM, SGLang, or JAX) and engineers building custom inference engines who want to maximize the performance of their models on AMD Instinct and Radeon GPUs.

Highlights

• Broad Framework Integration: Serves as the default kernel backend for LLM inference on ROCm, integrated into vLLM, SGLang, and ATOM.
• Diverse Operator Support: Includes optimized kernels for attention (MHA, MLA, Paged Attention), Mixture-of-Experts (MoE), GEMM, normalization, and quantization.
• Significant Performance Gains: Demonstrates substantial speedups, such as up to 17x for MLA decode kernels and 14x for MHA prefill kernels.
• Multi-Backend Support: Leverages various backends including Triton and FlyDSL to optimize kernels across different AMD GPU architectures (CDNA and RDNA).