deepseek-ai/DeepEP

DeepEP – High‑Performance Communication for Mixture‑of‑Experts (MoE) Models

What it is – DeepEP (DeepEveryParallel) is a C++/Python library that provides ultra‑fast GPU‑to‑GPU communication primitives needed for large‑scale Mixture‑of‑Experts (MoE) training and inference. It implements expert‑parallel (EP) all‑to‑all kernels, plus experimental support for pipeline, context and data parallelism, and remote memory access (Engram). The kernels are JIT‑compiled at runtime with the companion DeepJIT tool, allowing them to be tuned for the exact GPU architecture in use.

Why it matters – MoE models split a neural‑network layer into many experts and route each token to a few of them. This yields huge model capacity but requires moving tensors between GPUs at every layer. Conventional NCCL collectives become a bottleneck. DeepEP’s EP kernels achieve high‑throughput, low‑latency all‑to‑all dispatch/combine, support FP8 dispatch, and can pre‑fetch expert weights and reduce redundant gradients over NVLink, enabling the “expert‑replication” techniques described in MoonEP and UltraEP.

Key features

• JIT‑compiled EP kernels via DeepJIT – compiled for the exact SM count and GPU model at runtime.
• NCCL‑Gin backend – lightweight device‑side API that can reuse existing NCCL communicators.
• Unified EPBuffer API – single object that handles dispatch, combine, weight prefetch, gradient reduction, and buffer allocation.
• Experimental primitives – Engram (RDMA‑backed remote memory), Pipeline Parallel (PP), Bucket collectives for context‑ and data‑parallel workloads.
• Support for NVIDIA Hopper+ GPUs and, via a separate repo, Huawei Ascend 950 NPUs.
• Analytical SM/QP count estimation – no auto‑tuning needed.
• Redundant‑expert support – lb_prefetch_weights / lb_reduce_grads for dynamic expert replication.

Typical workflow

1. Create an EPBuffer with MoE hyper‑parameters (hidden size, top‑k, number of experts, FP8 flag). The buffer estimates how many SMs to allocate for the all‑to‑all kernels.
2. Dispatch – call buffer.dispatch(...) to route tokens to the appropriate experts across all ranks. The call returns an EventOverlap that can be waited on later, allowing compute‑communication overlap.
3. Run expert GEMMs on the received tokens.
4. Combine – call buffer.combine(...) (or the backward helper) to reduce expert outputs back to the original ranks.
5. (Optional) Load‑balance – use lb_prefetch_weights and lb_reduce_grads to replicate heavily‑loaded experts across NVLink‑connected GPUs.

Installation snapshot

# clone and init submodule (DeepJIT)
git submodule update --init --recursive
# install matching NCCL package (example for CUDA 13)
python -m pip install "nvidia-nccl-cu13>=2.32.3" --no-deps
# build and install the wheel
bash install.sh

The build compiles a C++ extension against the local CUDA, NCCL and C++20 toolchain. No GPU is required at install time; kernels are compiled on first use.

When to use DeepEP

• Training or inference of large MoE models (e.g., GPT‑style with thousands of experts).
• Scenarios that need expert replication to balance load across GPUs.
• Systems with NVLink intra‑node links and optionally RDMA inter‑node networks.
• Projects that already use PyTorch 2.x and want a drop‑in replacement for the all‑to‑all communication layer.

Limitations / experimental parts

• Engram, PP, and Bucket collectives are still marked experimental.
• EP kernels require GPU SMs; a pure RDMA‑only mode is not supported.
• Requires recent hardware (NVIDIA Hopper or newer) and CUDA 13.1+.

Resources

• Main repo: https://github.com/deepseek-ai/DeepEP
• DeepJIT submodule for runtime kernel compilation.
• Ascend‑compatible fork: https://github.com/deepseek-ai/DeepEP-Ascend
• Related load‑balancing projects: MoonshotAI/MoonEP, Dots‑Infra/UltraEP.

---

DeepEP is a specialized communication library for experts‑parallel deep‑learning workloads, offering JIT‑compiled kernels and APIs that let researchers and engineers scale MoE models efficiently across modern GPU clusters.