Dao-AILab/quack

A Quirky Assortment of CuTe Kernels

What it solves

QuACK provides a collection of high-performance GPU kernels for common deep learning operations, allowing developers to achieve "speed-of-light" performance for memory-bound kernels directly from Python using the CuTe-DSL.

How it works

The project implements various mathematical operations (kernels) using the CuTe-DSL, a domain-specific language for NVIDIA GPUs. It supports modern NVIDIA hardware including H100, B200/B300, and RTX 50 GPUs, and provides optional bindings for JAX.

Who it’s for

It is designed for researchers and engineers working with high-end NVIDIA GPUs who need optimized GPU kernels for operations like normalization and matrix multiplication (GEMM) without writing low-level CUDA C++.

Highlights

  • Optimized kernels for RMSNorm, Softmax, Cross Entropy, and LayerNorm (both forward and backward passes).
  • Specialized GEMM and epilogue implementations for Hopper and Blackwell architectures.
  • Integration with JAX via optional bindings.
  • Support for the latest CUDA toolkit (12.9+) and Python 3.12.

Related

  • Project
    NVIDIA/cutlass

    A collection of CUDA C++ and Python DSL abstractions for implementing high-performance matrix-matrix multiplication (GEMM) and the related linear algebra operations on NVIDIA GPUs.

  • Project
    NVIDIA/cudnn-frontend

    NVIDIA’s cuDNN Frontend is an open‑source, header‑only C++ API plus Python package that wraps the cuDNN Graph API. It provides easy‑to‑use graph construction, autotuning, and a catalog of open‑source high‑performance kernels (Flash‑Attention, fused GEMM + SwiGLU, block‑sparse attention, etc.) for Hopper/Blackwell GPUs. Install via pip or include the header; integrates with PyTorch and lets researchers and engineers get backend‑level speed without writing CUDA kernels from scratch.

  • Project
    MoonshotAI/FlashKDA

    A high-performance CUDA kernel implementation of Kimi Delta Attention (KDA) built on CUTLASS for NVIDIA SM90+ GPUs, serving as an optimized backend for the flash-linear-attention library.

  • Project
    lattice/quda

    A GPU-accelerated library for lattice QCD calculations, providing optimized Dirac operators and solvers for various fermion actions on NVIDIA hardware.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance tensor core kernel library for NVIDIA GPUs that provides optimized GEMM, MoE, and MQA kernels to accelerate LLM training and inference.