deepseek-ai/DeepGEMM

DeepGEMM: clean and efficient BLAS kernel library on GPU

What it solves

DeepGEMM is a high-performance CUDA kernel library designed to optimize the core computational primitives of modern large language models (LLMs). It addresses the need for efficient matrix multiplication (GEMM) and Mixture-of-Experts (MoE) operations, providing performance that matches or exceeds expert-tuned libraries while remaining simpler and more accessible for learning GPU kernel optimization.

How it works

The library provides a unified codebase for various tensor core kernels, including support for FP8, FP4, and BF16 precisions. It uses a system called DeepJIT to compile kernels at runtime, eliminating the need for CUDA compilation during installation. Key features include:

  • Diverse GEMM Support: Implements non-grouped, contiguous grouped, and masked grouped GEMMs tailored for different LLM phases (training, prefilling, and decoding).
  • Mega MoE: A fused mega-kernel that overlaps NVLink communication with tensor core computation, combining EP dispatch, linear layers, SwiGLU, and EP combine into one operation.
  • MQA Kernels: Specialized scoring kernels for the lightning indexer used in DeepSeek v3.2.
  • Hardware Optimization: Specifically optimized for NVIDIA SM90 and SM100 architectures.

Who it’s for

DeepGEMM is intended for developers and researchers working on LLM inference and training infrastructure, specifically those using NVIDIA H800 or similar high-end GPUs (SM90/SM100) who need maximum throughput and low-latency MoE operations.

Highlights

  • Extreme Performance: Achieves up to 1550 TFLOPS on H800 GPUs.
  • Runtime Compilation: Uses DeepJIT for on-the-fly kernel compilation to simplify installation.
  • Fused MoE: The Mega MoE kernel significantly reduces overhead by overlapping communication and computation.
  • Broad Precision Support: Supports FP8, FP4, and BF16 formats.
  • Simplified Design: Avoids heavy reliance on complex CUTLASS templates, making the codebase cleaner and easier to study.

Related

  • Project
    deepseek-ai/DeepEP

    DeepEP is a high‑performance communication library for Mixture‑of‑Experts training and inference. It provides JIT‑compiled all‑to‑all dispatch/combine kernels, a unified EPBuffer API, support for expert replication (weight prefetch & gradient reduction), and experimental pipeline/remote‑memory primitives. Designed for NVIDIA Hopper (and Ascend) GPUs with NVLink/RDMA, it lets large MoE models scale efficiently across multiple GPUs.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    pytorch/FBGEMM

    A repository of highly-optimized kernels for deep learning, providing low-precision matrix multiplication and convolution libraries for server-side inference and generative AI.

  • Project
    NVIDIA/TileGym

    A CUDA Tile kernel library providing tutorials and examples for tile-based GPU programming to optimize deep learning operators and LLM inference.

  • Project
    deepseek-ai/DeepGEMM-Ascend

    DeepGEMM‑Ascend is an open‑source, Ascend‑NPU‑specific port of the DeepGEMM matrix‑multiply library. It offers API‑compatible BF16/FP8/FP4 GEMM, MQA, and MoE kernels, hides Ascend’s low‑level layout complexities, and claims 98‑99 % hardware utilisation on Ascend 950. Installable via pip after building a C++ extension, it’s aimed at AI engineers who need fast linear‑algebra primitives on Huawei’s hardware.