facebookincubator/AITemplate

What it solves

AITemplate is designed to maximize inference speed for deep neural networks on NVIDIA and AMD GPUs. It addresses the performance bottlenecks of general-purpose runtimes by transforming models into highly optimized, self-contained C++ code, allowing for inference serving that reaches close to the hardware's theoretical maximum performance (roofline).

How it works

The framework converts neural network graphs into CUDA or HIP C++ code. It employs several advanced fusion techniques to reduce memory overhead and increase GPU utilization:

• Horizontal Fusion: Combines parallel operators (like GEMM or LayerNorm) with different input shapes into a single kernel.
• Vertical Fusion: Merges a range of operations (such as elementwise ops and reductions) directly into TensorCore or MatrixCore operations.
• Memory Fusion: Integrates memory operations like concatenation, split, and slice into the operators that precede them.

It includes a tool called FX2AIT, which converts PyTorch models into the AITemplate engine. If a model contains operators not yet supported by AITemplate, FX2AIT can perform partial acceleration by splitting the model.

Who it’s for

Developers and engineers who need extreme inference performance on Ampere-generation NVIDIA GPUs (SM80+) or CDNA2 AMD GPUs (MI-210/250), particularly for models like BERT, Stable Diffusion, and Vision Transformers.

Highlights

• Hardware Agnostic: Works across both NVIDIA and AMD GPU platforms.
• Zero Dependency: Compiled models are self-contained binaries that do not require third-party libraries like cuBLAS, cuDNN, or TensorRT at runtime.
• PyTorch Integration: Can use PyTorch tensors as inputs and outputs without extra memory copying, but can also run in environments entirely without PyTorch.
• Extensible: New operators or fused kernels can be added by defining a graph node and a backend codegen in Python.