66RING/tiny-flash-attention

What it solves

FlashAttention is a fast and memory-efficient attention algorithm used in modern machine learning, but the original implementation is complex. This project breaks it down into small, readable versions in several programming languages so learners can understand how it works instead of treating it as a black box.

How it works

The repository provides tiny implementations of FlashAttention in:

• Pure Python (naive) – the simplest version to follow the algorithm step by step.
• Triton – a GPU-oriented Python-like language, showing how to express the same idea for parallel hardware.
• C – a naive C implementation of the core attention computation.
• CUDA – a standalone CUDA kernel, plus a version wrapped as a Python binding.
• CUTLASS – a CUDA implementation using the CUTLASS library, with detailed tutorials in English and Chinese.
• Rust – listed but not yet completed (marked with an empty checkbox).

All code is kept minimal and focused on clarity rather than performance. The CUTLASS version includes step-by-step tutorials explaining the implementation on the author’s environment (CUTLASS v3.4, PyTorch 1.14, CUDA 12.4).

Who it’s for

Developers and students who want to learn how FlashAttention is implemented in different programming environments, especially those studying GPU programming, CUDA, or performance-oriented attention mechanisms.

Highlights

• Covers multiple languages: Python, Triton, C, CUDA, CUTLASS, and Rust (WIP).
• Each version is intentionally small and educational, not optimized for production.
• Includes standalone CUDA code and a Python-binding CUDA version.
• Provides detailed CUTLASS/CuTe tutorials in both English and Chinese.
• The project is still a work in progress, so more versions may be added.