microsoft/Tutel

What is Tutel?

Tutel is an open‑source library from Microsoft that implements a high‑performance Mixture‑of‑Experts (MoE) layer for large language models. It focuses on speeding up both training and inference of massive MoE models (hundreds of billions to a trillion parameters) by providing:

• Dynamic parallelism – the ability to switch between data‑parallel, expert‑parallel, and model‑parallel modes without extra cost.
• Specialized low‑precision kernels – support for NVIDIA FP4/FP8, AMD MXFP4, and block‑wise FP8 formats, which let very large models run on a single GPU or a modest GPU cluster.
• Fast gating operations – custom CUDA kernels for the routing step used by MoE models such as DeepSeek‑V3, Qwen‑3, Kimi, GLM‑5.x, and OpenAI‑GPT‑OSS.
• Multi‑node, multi‑GPU launchers – wrappers around torch.distributed and MPI that handle all‑to‑all communication efficiently.

The README shows concrete performance numbers (e.g., > 100 tokens‑per‑second for a 671 B DeepSeek model on 8 × MI300 GPUs) and provides ready‑to‑run Docker images for Azure GPU instances.

---

Who might use it?

• Research labs or companies that train or serve MoE‑based LLMs and need to squeeze the most throughput out of NVIDIA A100/H100, AMD MI300, or similar GPUs.
• Engineers building inference services (chatbots, code assistants, multimodal agents) who want to serve models like GLM‑5, Kimi‑K3, DeepSeek‑V3, Qwen‑3, or GPT‑OSS with low latency and high token‑throughput.
• Developers of custom MoE architectures who need a reference implementation of fast gating kernels and flexible parallelism switches.

---

How to get started (quick‑start guide)

1. Install PyTorch (CUDA ≥ 11.7 for NVIDIA, ROCm ≥ 6.2.2 for AMD) – the README provides the exact pip commands.
2. Install Tutel either directly from the repo:

   pip install -U git+https://github.com/microsoft/tutel@main
   

   or by cloning and running setup.py.
3. Run a sanity‑check example on a single GPU:

   python -m tutel.examples.helloworld --batch_size=16
   

   This exercises the optimized MoE layer and prints timing info.
4. Scale out:
   • For 8 GPUs on one machine:

     torchrun --nproc_per_node=8 -m tutel.examples.helloworld --batch_size=16
     

   • For multi‑node clusters, use the provided MPI launcher:

     mpiexec -bind-to none -host <hosts> -x LOCAL_SIZE=8 \
       python -m tutel.launcher.run -m tutel.examples.helloworld_ddp_tutel --batch_size=16
     

5. Serve a real model – pull a model checkpoint with huggingface_hub (e.g., moonshotai/Kimi-K3) and start the Docker container supplied in the README:

   docker run -e WORKER=1 -e LOCAL_SIZE=8 -p 8000:8000 \
     tutelgroup/deepseek-671b:mi300x8-chat-20260929 \
     --serve=core --try_path moonshotai/Kimi-K3 --max_seq_len 200000
   

   The container exposes a simple HTTP /chat endpoint that you can query with curl or integrate into your own UI.

---

When to consider alternatives

Situation	Why Tutel may not be ideal
You need CPU‑only inference	Tutel’s performance gains rely on GPU kernels; CPU support is minimal.
Your model is non‑MoE (dense transformer)	Tutel’s optimizations target the routing and expert‑parallel parts of MoE; a dense model won’t benefit.
You run on older GPUs (e.g., < A100, no ROCm support)	The low‑precision kernels (FP4/FP8) require recent hardware; you’d fall back to standard FP16/FP32 with little speedup.
You prefer a single‑library, end‑to‑end training framework (e.g., DeepSpeed, Megatron‑LM)	While Tutel can be combined with those, it is primarily a MoE kernel/launcher layer; you may need extra glue code.

If any of the above match your constraints, look at alternatives such as DeepSpeed MoE, Megatron‑LM, or FasterTransformer.

---

TL;DR

Tutel is a Microsoft‑maintained, GPU‑focused library that makes large MoE language models run faster by providing dynamic parallelism, low‑precision kernels, and ready‑to‑use Docker images. It’s best suited for teams serving or training trillion‑parameter MoE models on modern NVIDIA or AMD GPUs.