microsoft/Tutel

Tutel MoE: Optimized Mixture-of-Experts Library, Support GptOss/DeepSeek/Kimi-K2/Qwen3 using FP8/NVFP4/MXFP4

What it solves

Tutel is an optimized implementation of Mixture-of-Experts (MoE) designed to accelerate both training and inference for large-scale models. It addresses the computational bottlenecks and memory constraints associated with MoE architectures, particularly when deploying massive models like DeepSeek, Kimi, and GLM on high-end GPUs (NVIDIA A100/H100/B200 and AMD MI300/MI325/MI355).

How it works

Tutel provides a highly optimized MoE layer and a parallel solution called "No-penalty Parallelism/Sparsity/Capacity Switching," which allows for dynamic adjustments to parallelism, sparsity, and capacity without performance penalties. It supports advanced quantization formats like NVFP4, MXFP4, and BlockwiseFP8 to reduce memory footprints and increase throughput. The project also includes specialized gating APIs (e.g., for Qwen3 and DeepSeek) and utilizes efficient communication primitives like NCCL all-to-all for distributed processing across multiple GPUs and nodes.

Who it’s for

It is intended for AI researchers and engineers working with massive MoE-based LLMs who need to maximize throughput and minimize latency on NVIDIA and AMD hardware, as well as those implementing custom MoE layers from scratch.

Highlights

  • Broad Hardware Support: Optimized for NVIDIA A100/H100/B200 and AMD MI300/MI325/MI355 GPUs.
  • High-Performance Inference: Supports massive models including DeepSeek-V3.2, Kimi-K3, and GLM-5.x with high token-per-second (TPS) rates.
  • Dynamic Configuration: Allows free-cost switching of parallelism (DP, EP, MP), top-k sparsity, and capacity volume during runtime.
  • Advanced Quantization: Direct support for NVFP4 and MXFP4 inference to fit trillion-parameter models into available VRAM.
  • Long-Context Support: Capable of handling context sizes up to 1 million tokens for specific models like GLM-5.x and Kimi-K3.

Related

  • Dispatch
    Mixture of Experts (MoE) Explained

    Mixture of Experts (MoE) allows transformer models to scale parameters while maintaining efficient pretraining and faster inference by activating only a subset of neural network experts per token.

  • Project
    cursor/mixture-of-kittens

    A deterministic MoE training megakernel for NVIDIA Blackwell NVL72s that fuses computation and communication to eliminate CPU-GPU synchronization and accelerate training.

  • Project
    EfficientMoE/MoE-Infinity

    MoE-Infinity is a library for cost-effective MoE inference that enables running large Mixture-of-Experts models on memory-constrained GPUs by offloading expert weights to host memory and SSD.

  • Project
    kvcache-ai/ktransformers

    A framework for efficient LLM inference and fine-tuning using CPU-GPU heterogeneous computing, specifically optimized for ultra-large MoE models on limited hardware.

  • Project
    Dao-AILab/sonic-moe

    A high-performance Mixture-of-Experts (MoE) implementation optimized for NVIDIA Hopper and Blackwell GPUs to reduce memory usage and increase training throughput.