cursor/mixture-of-kittens

Mixture-of-experts (MoE) training megakernel for NVL72s

What it solves

Mixture-of-Kittens (MoK) addresses the efficiency bottlenecks in training Mixture-of-Experts (MoE) models on high-end NVIDIA Blackwell NVL72 systems. It specifically targets the overhead caused by CPU-GPU synchronization and the separation of computation and communication during MoE training, which often slows down the training process.

How it works

MoK is a deterministic "megakernel" that fuses all MoE computation and communication into a single CUDA kernel. By doing this, it overlaps compute and inter-GPU networking at a configurable granularity and completely eliminates the need for CPU-GPU synchronization. It supports both BF16 and MXFP8 precision formats and handles both forward and backward passes.

Who it’s for

This project is designed for researchers and engineers working with NVIDIA Blackwell GPUs (SM100 or SM103) and large-scale MoE training, such as those training production-grade LLMs.

Highlights

  • High Performance: Up to 2.37x faster for MXFP8 forward and 1.92x faster for BF16 forward compared to the fastest baselines.
  • Fused Kernel: Combines computation and communication into one operation to maximize hardware utilization.
  • Precision Support: Supports BF16 and MXFP8 for optimized training.
  • Two-Layer API: Provides a low-level "Ops layer" for direct CUDA kernel calls and a high-level "Functional layer" for easier production integration.

Related

  • Project
    microsoft/Tutel

    An optimized Mixture-of-Experts (MoE) implementation that accelerates training and inference for massive LLMs on NVIDIA and AMD GPUs using advanced parallelism and quantization.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance tensor core kernel library for NVIDIA GPUs that provides optimized GEMM, MoE, and MQA kernels to accelerate LLM training and inference.

  • Project
    Dao-AILab/sonic-moe

    A high-performance Mixture-of-Experts (MoE) implementation optimized for NVIDIA Hopper and Blackwell GPUs to reduce memory usage and increase training throughput.

  • Project
    deepseek-ai/DeepEP

    A high-performance communication library for MoE models that provides optimized GPU kernels for expert parallelism to maximize throughput and minimize SM resource usage.

  • Project
    InternLM/xtuner

    A next-generation LLM training engine optimized for ultra-large-scale MoE models up to 1T parameters, supporting multimodal training and advanced reinforcement learning.