deepseek-ai/DeepEP

DeepEP: an efficient expert-parallel communication library

What it solves

DeepEP addresses the communication bottlenecks in modern machine learning training and inference, specifically for Mixture-of-Experts (MoE) models. It provides high-throughput, low-latency all-to-all GPU kernels for expert parallelism (EP) dispatch and combine operations, reducing the overhead of moving tokens between GPUs while minimizing the consumption of Streaming Multiprocessors (SMs).

How it works

DeepEP implements a high-performance communication library that utilizes a lightweight Just-In-Time (JIT) compilation module to generate kernels at runtime, removing the need for CUDA compilation during installation. It uses the NCCL Gin backend for lightweight communication and provides an ElasticBuffer interface to unify high-throughput and low-latency APIs. The library supports low-precision formats like FP8 and leverages RDMA and NVLink for efficient internode and intranode data transfer. It also includes experimental primitives for pipeline parallelism (PP), context parallelism (CP), and remote memory access (Engram).

Who it’s for

It is designed for developers and researchers training or deploying large-scale MoE models on Hopper (SM90) GPUs or newer architectures who need to maximize hardware bandwidth and minimize SM resource usage for communication.

Highlights

  • Extreme Efficiency: V2 achieves up to 1.3x peak performance over V1 while using up to 4x fewer SM resources.
  • JIT Compilation: Kernels are compiled at runtime, simplifying installation and deployment.
  • Analytical Tuning: Automatically calculates optimal SM and QP counts, eliminating the need for manual auto-tuning.
  • Broad Parallelism Support: Beyond Expert Parallelism, it offers experimental support for Pipeline Parallelism, Context Parallelism, and remote memory access.
  • Low Precision Support: Optimized for FP8 dispatching and BF16 combining.

Related

  • Project
    MoonshotAI/MoonEP

    A communication library for Expert Parallelism that eliminates load imbalance in MoE models by using dynamic redundant experts and zero-copy communication.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance tensor core kernel library for NVIDIA GPUs that provides optimized GEMM, MoE, and MQA kernels to accelerate LLM training and inference.

  • Project
    microsoft/Tutel

    An optimized Mixture-of-Experts (MoE) implementation that accelerates training and inference for massive LLMs on NVIDIA and AMD GPUs using advanced parallelism and quantization.

  • Project
    EfficientMoE/MoE-Infinity

    MoE-Infinity is a library for cost-effective MoE inference that enables running large Mixture-of-Experts models on memory-constrained GPUs by offloading expert weights to host memory and SSD.

  • Project
    Tencent/hpc-ops

    A production-grade operator library for LLM inference that provides highly optimized CUDA kernels for attention, MoE, and sampling to reduce serving latency.