mlc-ai/pith-train

Compact and Agent-Native MoE Training System

What it solves

Production MoE (Mixture-of-Experts) training frameworks are often overly complex, relying on heavy compiled extensions and indirect design patterns that make them difficult for both humans and AI coding agents to modify or optimize. PithTrain provides a high-performance training system that is compact enough to be read end-to-end by an agent or human while maintaining production-grade capabilities.

How it works

It is an agent-native framework structured into three layers: an Application layer for training loops, an Engine layer for model protocols and distributed training, and an Operator layer for low-level kernels. It implements a 4D parallelism strategy (Pipeline, Data, Context, and Expert parallelism) and utilizes a DualPipeV scheduler for overlapped forward-backward execution and P2P communication. The system supports FP8 training and integrates with libraries like DeepGEMM and FlashAttention.

Who it’s for

Researchers and developers who need to train MoE models (such as Qwen or DeepSeek architectures) and want a codebase that is easy to navigate, modify, and extend, especially when using AI coding assistants.

Highlights

  • Agent-Native Design: A compact codebase of approximately 11K lines of Python with minimal implicit indirection.
  • Production Performance: Supports 4D parallelism and compute-communication overlap.
  • FP8 Training: Built-in support for FP8 linear and quantization for efficiency.
  • Flexible Checkpointing: Supports reshardable checkpoints and conversion to Hugging Face format.

Related

  • Dispatch
    Mixture of Experts (MoE) Explained

    Mixture of Experts (MoE) allows transformer models to scale parameters while maintaining efficient pretraining and faster inference by activating only a subset of neural network experts per token.

  • Project
    microsoft/Tutel

    Tutel is Microsoft’s open‑source, high‑performance MoE library for large language models. It provides dynamic parallelism, low‑precision inference (NVFP4, MXFP4, FP8, BF16), support for up to 1 M‑token contexts, vision extensions, and Docker images that serve models like DeepSeek‑V3.2, Kimi‑K3, GLM‑5.x, Qwen‑3, and GPT‑OSS on both NVIDIA and AMD GPUs. Install via pip from the Git repo or build from source, then run the supplied Docker containers or use the Python API for custom MoE routing.

  • Project
    deepseek-ai/DeepEP

    A high-performance communication library for MoE models that provides optimized GPU kernels for expert parallelism to maximize throughput and minimize SM resource usage.

  • Project
    PaddlePaddle/ERNIE

    A family of large-scale multimodal MoE models and development toolkits providing state-of-the-art performance in text and vision-language understanding.

  • Dispatch
    Qwen1.5-MoE-A2.7B Release Notes

    Qwen introduces Qwen1.5-MoE-A2.7B, a Mixture-of-Experts model that matches the performance of 7B dense models while using only 2.7 billion activated parameters.