MoonshotAI/MoonEP

MoonEP: A Perfectly Balanced Expert Parallelism Library via Dynamic Redundant Experts

What it solves

MoonEP addresses the load imbalance problem in Mixture-of-Experts (MoE) models during Expert Parallelism (EP). In traditional MoE, some experts receive significantly more tokens than others, leading to "hot ranks" that slow down the entire training or inference process and can cause GPU memory fragmentation or Out-of-Memory (OOM) errors.

How it works

MoonEP ensures every GPU rank receives an identical number of tokens regardless of the routing distribution. It achieves this through:

  • Dynamic Redundant Experts: It plans a small number of redundant experts online and prefetches them to ranks that need them, ensuring the workload is perfectly balanced across all ranks.
  • Online Planning: A high-performance GPU kernel handles the planning of these redundant experts with minimal overhead.
  • Zero-Copy Communication: It uses fused permute/unpermute operations to send tokens directly to their final positions on remote ranks, providing views of the communication buffer directly to the computation, which eliminates expensive data copies.
  • Static Memory Shapes: By maintaining a fixed buffer size for tokens, it prevents memory fragmentation and ensures predictable memory usage.

Who it’s for

Developers and researchers building large-scale MoE models who need to optimize communication overhead and maintain stable training performance across multiple GPUs using NVLink.

Highlights

  • Immune to Imbalance: Iteration time remains flat even as routing imbalance increases.
  • Zero-Copy Efficiency: Faster raw communication by eliminating the copy between communication and user buffers.
  • Static Memory: Prevents GPU memory fragmentation and OOMs at high imbalance levels.
  • Integrated Weight Prefetching: Includes specialized kernels for prefetching and reducing gradients for redundant experts.

Related

  • Project
    deepseek-ai/DeepEP

    DeepEP is a high‑performance communication library for Mixture‑of‑Experts training and inference. It provides JIT‑compiled all‑to‑all dispatch/combine kernels, a unified EPBuffer API, support for expert replication (weight prefetch & gradient reduction), and experimental pipeline/remote‑memory primitives. Designed for NVIDIA Hopper (and Ascend) GPUs with NVLink/RDMA, it lets large MoE models scale efficiently across multiple GPUs.

  • Dispatch
    Mixture of Experts (MoE) Explained

    Mixture of Experts (MoE) allows transformer models to scale parameters while maintaining efficient pretraining and faster inference by activating only a subset of neural network experts per token.

  • Dispatch
    vLLM Elastic Expert Parallelism

    vLLM introduces Elastic Expert Parallelism (Elastic EP), enabling Mixture-of-Experts (MoE) deployments to scale the number of workers up or down at runtime without requiring a server restart.

  • Project
    EfficientMoE/MoE-Infinity

    MoE‑Infinity is a library that enables inference of large Mixture‑of‑Experts language models on GPUs with limited VRAM by off‑loading expert weights to CPU/SSD and using an activation‑aware cache, plus a production‑grade OpenAI‑compatible server.

  • Dispatch
    Mixture of Experts (MoEs) in Transformers

    Hugging Face has redesigned the transformers library to make Mixture of Experts (MoEs) first-class citizens through a new weight loading refactor, a pluggable expert backend, and native expert parallelism.