openxla/tokamax

What it solves

Tokamax provides high-performance, custom accelerator kernels for NVIDIA GPUs and Google TPUs, reducing the need for developers to manually implement low-level optimizations for common deep learning operations. It specifically targets the performance gap in operations like attention and normalization that are critical for modern AI models.

How it works

Built on top of JAX and Pallas, Tokamax implements state-of-the-art kernels for specific operations. It allows users to select specific implementations (e.g., Triton or Mosaic) or let the library automatically choose the most efficient one based on the kernel shape. To maximize performance, it includes an autotuning system that searches for the best configuration for a given workload, which can then be serialized and reused to ensure numerical consistency.

Who it’s for

It is designed for AI researchers and engineers working with JAX who need to squeeze maximum performance out of GPU and TPU hardware through custom kernels and autotuning.

Highlights

• Multi-Accelerator Support: Works across both NVIDIA GPUs and Google TPUs.
• Pre-built Kernels: Includes optimized implementations of FlashAttention, Gated Linear Units (SwiGLU), LayerNorm, RMSNorm, and Mixture of Experts (ragged_dot).
• Autotuning Tooling: Provides a framework to automatically find the fastest kernel configurations and allows users to define their own autotuning search spaces.
• Hardware Benchmarking: Includes utilities to measure actual accelerator execution time, bypassing JAX Python overhead, with support for tools like the CUPTI profiler.