NVIDIA/Megatron-LM

Ongoing research training transformer models at scale

What it solves

Megatron-LM addresses the challenge of training massive transformer models (ranging from 2 billion to 462 billion parameters) across thousands of GPUs. It provides the necessary infrastructure to handle the memory and computational demands of large-scale model training while maximizing hardware efficiency.

How it works

The project is split into two main parts: Megatron Core, a composable library of GPU-optimized building blocks (kernels, transformer components, and optimizers), and Megatron-LM, a reference implementation that combines these blocks with pre-configured training scripts for easier experimentation.

To achieve scale, it employs several advanced parallelism strategies:

  • Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), Expert Parallelism (EP), and Context Parallelism (CP).
  • It supports various mixed-precision formats to optimize memory and speed, including FP16, BF16, FP8, and FP4.
  • It includes specialized support for Mixture-of-Experts (MoE) architectures and hybrid architectures like Falcon-H1 (Transformer-Mamba).

Who it’s for

  • ML Engineers and Framework Developers who need a high-performance, composable library to build custom training pipelines.
  • Research Teams looking for a reference implementation to quickly experiment with and train large-scale language models.

Highlights

  • Extreme Scalability: Capable of training models up to 462B parameters across 6,144 H100 GPUs.
  • High Hardware Efficiency: Achieves up to 47% Model FLOP Utilization (MFU) on H100 clusters.
  • Broad Architecture Support: Includes implementations for DeepSeek-V4, MoE models, and hybrid Transformer-Mamba architectures.
  • Interoperability: Offers bidirectional checkpoint conversion between Hugging Face and Megatron via Megatron Bridge.

Related

  • Dispatch
    Training Language Models with Megatron-LM

    Hugging Face provides a guide on using NVIDIA's Megatron-LM framework to efficiently pre-train large language models on GPUs, including integration with the Transformers library.

  • Project
    deepspeedai/DeepSpeed

    A deep learning optimization library that enables the training of models with trillions of parameters by optimizing memory and compute efficiency across distributed systems.

  • Project
    ServiceNow/Fast-LLM

    A high-performance library for training large language models that uses 3D parallelism and optimized kernels to reduce training time and cost.

  • Project
    NVIDIA-NeMo/Megatron-Bridge

    A PyTorch-native library that enables bidirectional checkpoint conversion and high-performance training between Hugging Face and NVIDIA Megatron Core.

  • Project
    InternLM/xtuner

    A next-generation LLM training engine optimized for ultra-large-scale MoE models up to 1T parameters, supporting multimodal training and advanced reinforcement learning.