Training Language Models with Megatron-LM

Overview

Megatron-LM is a high-performance framework developed by NVIDIA's Applied Deep Learning Research team designed for the efficient pre-training of large transformer models on GPUs. While more complex to set up than the Hugging Face Trainer API or accelerate library, Megatron-LM provides significant speedups through specialized data handling and hardware-level optimizations.

Technical Optimizations in Megatron-LM

Megatron-LM achieves superior training efficiency compared to standard PyTorch loops through two primary mechanisms: optimized data loading and kernel fusion.

Efficient DataLoader

Megatron-LM utilizes a DataLoader that tokenizes and shuffles data before training begins. It computes numbered sequences with indexes once and determines the number of epochs based on training parameters. This approach avoids the need to iterate through the entire dataset before repeating for a new epoch, which results in a smoother learning curve and reduced training time.

Fused CUDA Kernels

To minimize memory overhead, Megatron-LM employs fused CUDA kernels. In standard PyTorch operations, data is fetched from memory, computed, and saved back for every discrete operation. Kernel fusion combines similar operations into a single hardware operation, allowing intermediary results to be stored in GPU registers rather than being copied back to main memory.

Additionally, Megatron-LM uses a fused implementation of AdamW from the NVIDIA Apex library, which offers faster performance than the standard PyTorch implementation.

Implementation Workflow

Training a model (such as GPT-2) in Megatron-LM involves a four-step pipeline: environment setup, data preprocessing, training, and model conversion.

Environment Setup

The recommended setup is using an NVIDIA PyTorch Container from NGC, which includes all necessary dependencies. Manual installations require the latest versions of PyTorch, CUDA, NCCL, NVIDIA Apex, and the nltk library. Users must also provide the tokenizer's vocab.json and merges.txt files within the Megatron-LM directory.

Data Preprocessing

Data must be converted into a loose JSON format (one sample per line) before being processed. Megatron-LM then tokenizes, shuffles, and converts this data into a binary format using tools/preprocess_data.py. This process generates .idx and .bin files, which are required for the training phase. The dataset-impl can be configured as 'lazy', 'cached', or 'mmap'.

Training Execution

Training is launched via torch.distributed.launch. For a 110M parameter model (like CodeParrot-small), training on 8 GPUs takes approximately 12 hours with the following configuration:

  • Architecture: 12 layers, 768 hidden size, 12 attention heads.
  • Sequence Length: 1024.
  • Optimizer: AdamW with a cosine learning rate decay.
  • Batch Size: Micro-batch size of 12 and global batch size of 192.

Model Parallelism Strategies

For models too large to fit on a single GPU, Megatron-LM supports two types of model parallelism:

  1. Tensor Parallelism: Splits the execution of a single transformer module across multiple GPUs via the tensor-model-parallel-size parameter.
  2. Pipeline Parallelism: Splits transformer modules into equally sized stages across GPUs via the pipeline-model-parallel-size parameter.

Integration with Hugging Face Transformers

To use a Megatron-LM trained model for evaluation or production, it must be converted to a format supported by the transformers library. This is achieved by converting the model_optim_rng.pt checkpoint file into a pytorch_model.bin file using the convert_megatron_gpt2_checkpoint.py script.

Once converted, the model can be loaded using AutoModelForCausalLM. For very large models trained with model parallelism, the device_map="auto" argument can be used to automatically dispatch weights across available GPUs and CPU RAM via the accelerate library.

Framework Selection Guidance

Megatron-LM is best suited for pre-training large models or performing extended fine-tuning due to its high optimization. However, it introduces overhead in preprocessing and conversion steps. For shorter fine-tuning of medium-sized models, the Hugging Face Trainer API and accelerate library are recommended as they are device-agnostic and offer greater flexibility.

Sources