NVIDIA/Megatron-LM

Ongoing research training transformer models at scale

What it solves

Megatron-LM provides a GPU-optimized framework for training massive transformer models (from 2 billion to 462 billion parameters) across thousands of GPUs. It addresses the memory and computational bottlenecks of training large-scale language models by implementing advanced distributed training strategies.

How it works

The project is split into two main components:

  • Megatron Core: A composable library of GPU-optimized building blocks. It provides the essential transformer components and supports multiple parallelism strategies, including Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), Expert Parallelism (EP), and Context Parallelism (CP). It also supports mixed precision training (FP16, BF16, FP8, and FP4).
  • Megatron-LM: A reference implementation that combines Megatron Core with pre-configured training scripts for rapid experimentation and research.

Who it’s for

  • ML Engineers: Those building custom training pipelines for large-scale models.
  • Research Teams: Teams conducting research on distributed training and transformer architectures.
  • Framework Developers: Developers creating their own training frameworks using optimized GPU kernels.

Highlights

  • Extreme Scalability: Capable of training models up to 462B parameters on clusters of up to 6,144 H100 GPUs.
  • Detailed Parallelism: Comprehensive support for TP, PP, DP, EP, and CP to optimize GPU utilization.
  • High Efficiency: Achieves up to 47% Model FLOP Utilization (MFU) on H100 clusters.
  • Crossover Support: Includes Megatron Bridge for bidirectional checkpoint conversion between Hugging Face and Megatron formats.
  • Modern Architecture Support: Initial support for DeepSeek-V4 and Falcon-H1 hybrid transformer-Mamba architectures.

Related

  • Dispatch
    Training Language Models with Megatron-LM

    Hugging Face provides a guide on using NVIDIA's Megatron-LM framework to efficiently pre-train large language models on GPUs, including integration with the Transformers library.

  • Project
    deepspeedai/DeepSpeed

    A deep learning optimization library that enables the training of trillion-parameter models through memory innovations like ZeRO and 3D-Parallelism.

  • Project
    ServiceNow/Fast-LLM

    A high-performance library for training large language models that uses 3D parallelism and optimized kernels to reduce training time and cost.

  • Project
    verl-project/verl

    A flexible and efficient RL training library for LLMs that simplifies the implementation of RLHF algorithms and scales to trillion-parameter models.

  • Project
    baidu-baige/LoongForge

    An open-source training framework that accelerates the training of LLMs, VLMs, diffusion, and embodied models using a multi-backend architecture.