Dao-AILab/sonic-moe

Accelerating MoE with IO and Tile-aware Optimizations

What it solves

SonicMoE addresses the performance bottlenecks of Mixture-of-Experts (MoE) layers, specifically focusing on reducing activation memory usage and increasing training throughput on modern NVIDIA GPUs.

Related

  • Project
    microsoft/Tutel

    An optimized Mixture-of-Experts (MoE) implementation that accelerates training and inference for massive LLMs on NVIDIA and AMD GPUs using advanced parallelism and quantization.

  • Dispatch
    Mixture of Experts (MoE) Explained

    Mixture of Experts (MoE) allows transformer models to scale parameters while maintaining efficient pretraining and faster inference by activating only a subset of neural network experts per token.

  • Project
    EfficientMoE/MoE-Infinity

    MoE-Infinity is a library for cost-effective MoE inference that enables running large Mixture-of-Experts models on memory-constrained GPUs by offloading expert weights to host memory and SSD.

  • Project
    cursor/mixture-of-kittens

    A deterministic MoE training megakernel for NVIDIA Blackwell NVL72s that fuses computation and communication to eliminate CPU-GPU synchronization and accelerate training.

  • Project
    deepseek-ai/DeepEP

    A high-performance communication library for MoE models that provides optimized GPU kernels for expert parallelism to maximize throughput and minimize SM resource usage.