xtuner: a training engine for ultra-large MoE models supporting up to 1T parameters and multimodal training

What it solves

XTuner V1 is a training engine designed to handle the complexities of training ultra-large-scale Mixture-of-Experts (MoE) models. It addresses the high memory and computational demands of models reaching up to 1 trillion parameters, while simplifying the parallelism strategies required to scale these models across hardware.

How it works

XTuner V1 optimizes the training process through several key technical approaches:

• Dropless Training: It reduces the complexity of expert parallelism, allowing 200B-scale MoE models to be trained without it and 600B models to require only intra-node parallelism.
• Memory Optimization: It enables the training of 200B MoE models with 64k sequence lengths without needing sequence parallelism, while also supporting DeepSpeed Ulysses for linear scaling of sequence length.
• Hardware Acceleration: The engine is optimized for both NVIDIA GPUs (FP8/BF16) and Ascend NPUs (BF16), achieving high throughput that can surpass traditional 3D parallel schemes for models above 200B parameters.

Who it’s for

This tool is intended for AI researchers and engineers working with ultra-large MoE models, specifically those focusing on pre-training, instruction fine-tuning, and reinforcement learning (RL) at scale.

Highlights

• Massive Scale Support: Capable of training MoE models up to 1T parameters.
• Multimodal Capabilities: Full support for vision-language model pre-training and supervised fine-tuning.
• Advanced RL Algorithms: Implements Group Relative Policy Optimization (GRPO) and is expanding to include MPO and DAPO.
• Cross-Hardware Compatibility: Optimized for both NVIDIA H800 and Ascend A3 Supernodes.