mjun0812/flash-attention-prebuild-wheels

Provide with pre-build flash-attention 2 and 3 package wheels on Linux and Windows using GitHub Actions

flash‑attention‑prebuild‑wheels

What it is – A GitHub repo that hosts ready‑to‑install binary wheels for the flash‑attention library (FlashAttention 1, 2 and 3). FlashAttention is a high‑performance CUDA/ROCm implementation of the attention operation used in large language models. Building it from source is slow and requires a powerful GPU, so this project supplies pre‑compiled wheels for many combinations of Python, CUDA (or ROCm), PyTorch, and platform (Linux x86_64, Linux ARM64, Windows).

Why it matters – Researchers and developers can add FlashAttention to their Python environments with a single pip install command, avoiding hours of compilation and the need for a custom build environment. The repository covers 100 % of the listed version matrix (over 1 000 wheels) and even provides wheels for newer FlashAttention 3, which needs Hopper‑class GPUs and CUDA 12.3+.

How to use it

  1. Choose the desired wheel name, which encodes the FlashAttention version, CUDA/ROCm version, PyTorch version, and Python version, e.g.
    flash_attn-2.6.3+cu124torch2.5-cp312-cp312-linux_x86_64.whl
  2. Locate the exact URL on the Useful Search Page (https://mjunya.com/flash-attention-prebuild-wheels/), the doc/packages.md list, or the GitHub releases page.
  3. Install directly from GitHub:
    pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.0.0/flash_attn-2.6.3+cu124torch2.5-cp312-cp312-linux_x86_64.whl
    or download first and install locally.

Supported platforms – Linux x86_64, Linux ARM64, Windows, and Linux with AMD ROCm. Wheels are built on a mix of GitHub‑hosted runners, self‑hosted machines, and AWS CodeBuild, using many‑linux_2_28 containers for broad glibc compatibility.

When you might need to build your own – If a specific version combination is missing, you can fork the repo, edit create_matrix.py to add the desired matrix entry, tag a new release (e.g., v0.9.0), and let the CI build the wheel. Self‑hosted runners are required for very large builds that exceed GitHub‑hosted job limits.

Citation – The README provides a BibTeX entry for acknowledging the wheel repository in academic work.

All information is taken directly from the repository’s README.

Related

  • Project
    wildminder/AI-windows-whl

    A curated collection of pre-compiled Python wheels for AI/ML libraries that are difficult to install on Windows, eliminating the need to build from source.

  • Project
    MoonshotAI/FlashKDA

    A high-performance CUDA kernel implementation of Kimi Delta Attention (KDA) built on CUTLASS for NVIDIA SM90+ GPUs, serving as an optimized backend for the flash-linear-attention library.

  • Project
    fla-org/flash-linear-attention

    A hardware-efficient library of subquadratic kernels and layers for linear attention, state space models, and hybrid LLM architectures across multiple hardware backends.

  • Project
    meta-pytorch/attention-gym

    A collection of kernels, guides, and examples for PyTorch FlexAttention and other novel attention variants, including sparse and linear attention.

  • Project
    QwenLM/FlashQLA

    FlashQLA is a Python‑package kernel library that speeds up the Gated‑Delta‑Rule linear‑attention used in Qwen models. Built with TileLang, it offers 2‑3× faster forward and ~2× faster backward passes on modern NVIDIA GPUs, automatic intra‑card parallelism, and a drop‑in API compatible with flash‑linear‑attention.