Dao-AILab/sonic-moe

Accelerating MoE with IO and Tile-aware Optimizations

What it solves

SonicMoE addresses the performance bottlenecks of Mixture-of-Experts (MoE) layers, specifically focusing on reducing activation memory usage and increasing training throughput on modern NVIDIA GPUs.

Related

  • Project
    microsoft/Tutel

    Tutel is an optimized engine for Mixture-of-Experts models, supporting fast inference on NVIDIA and AMD GPUs with low-precision formats, dynamic configuration changes, and many popular large models.

  • Dispatch
    Mixture of Experts (MoE) Explained

    Mixture of Experts (MoE) allows transformer models to scale parameters while maintaining efficient pretraining and faster inference by activating only a subset of neural network experts per token.

  • Project
    EfficientMoE/MoE-Infinity

    MoE‑Infinity is a library that enables inference of large Mixture‑of‑Experts language models on GPUs with limited VRAM by off‑loading expert weights to CPU/SSD and using an activation‑aware cache, plus a production‑grade OpenAI‑compatible server.

  • Project
    cursor/mixture-of-kittens

    Mixture‑of‑Kittens (MoK) is a deterministic mixture‑of‑experts megakernel for NVIDIA Blackwell GPUs. It merges compute and inter‑GPU communication into a single CUDA kernel, supports BF16 and MXFP8, and delivers up to 2.37× speed‑ups over existing MoE implementations. Install via pip, configure a `MoKConfig`, create a symmetric‑memory workspace, build a schedule, and call the functional `forward`/`backward` APIs. Designed for large‑scale MoE training on NVL72 hardware.

  • Project
    deepseek-ai/DeepEP

    DeepEP is a high‑performance communication library for Mixture‑of‑Experts training and inference. It provides JIT‑compiled all‑to‑all dispatch/combine kernels, a unified EPBuffer API, support for expert replication (weight prefetch & gradient reduction), and experimental pipeline/remote‑memory primitives. Designed for NVIDIA Hopper (and Ascend) GPUs with NVLink/RDMA, it lets large MoE models scale efficiently across multiple GPUs.