ROCm/aiter

What it solves

AITER (AI Tensor Engine for ROCm) provides a high-performance library of optimized GPU kernels specifically for AMD GPUs. It solves the problem of framework developers needing production-ready, highly tuned operators for AI inference and training without having to write low-level GPU kernels from scratch.

How it works

It acts as a unified collection of operators that can be integrated into various AI frameworks. AITER utilizes multiple kernel backends to achieve maximum performance, including Triton, Composable Kernel (CK), and hand-tuned Assembly (ASM). It also leverages FlyDSL for specific operators like GEMM and MoE, and includes a lightweight C++ template library called Opus to accelerate the development and build time of HIP kernels.

Who it’s for

This library is primarily for framework developers (such as those working on vLLM, SGLang, or JAX) and engineers building custom AI inference engines who need to optimize workloads on AMD Instinct and Radeon GPUs.

Highlights

• Broad Framework Support: Serves as the default kernel backend for LLM inference on ROCm, integrated into vLLM, SGLang, and ATOM.
• Diverse Operator Set: Includes optimized kernels for attention (MHA, MLA, Paged Attention), Mixture-of-Experts (MoE), GEMM, normalization, and quantization.
• Multi-Backend Approach: Combines Triton, CK, and hand-tuned ASM to optimize for different hardware architectures.
• Significant Performance Gains: Demonstrates substantial speedups, such as up to 17x for MLA decode kernels and 14x for MHA prefill kernels.
• Flexible APIs: Offers both C++ and Python APIs for integration.