NVIDIA/Megatron-Energon

Megatron's multi-modal data loader

What it solves

Megatron Energon addresses the challenge of loading and managing massive amounts of multi-modal training data for large-scale models. It simplifies the process of blending diverse datasets and distributing data loading across large clusters of nodes and processes while maintaining reproducibility and resumability.

How it works

Energon acts as a multi-modal data loader that can be used as part of the Megatron-LM framework or as a standalone Python package. It supports WebDataset and JSONL formats. Users can use a command-line tool (energon) to prepare and lint datasets to ensure they are compatible with the system. Once prepared, the library provides functions like get_train_dataset and get_loader to feed data into training loops.

Who it’s for

It is designed for researchers and engineers training large multi-modal models who need a scalable, distributed data pipeline that can handle various types of data samples and processing.

Highlights

  • Cluster-scale distribution: Efficiently distributes data loading work across many nodes and processes.
  • Multi-modal support: Specifically built to handle diverse types of data samples.
  • Dataset blending: Ability to mix and blend multiple different datasets together.
  • Reliability: Ensures that training runs are reproducible and can be resumed from where they left off.

Related

  • Dispatch
    Training Language Models with Megatron-LM

    Hugging Face provides a guide on using NVIDIA's Megatron-LM framework to efficiently pre-train large language models on GPUs, including integration with the Transformers library.

  • Project
    texttron/tevatron

    A unified toolkit for training and deploying billion-scale neural retrievers across different languages and modalities using LLMs.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    InternLM/xtuner

    A next-generation LLM training engine optimized for ultra-large-scale MoE models up to 1T parameters, supporting multimodal training and advanced reinforcement learning.

  • Project
    NVIDIA/Megatron-LM

    A GPU-optimized library and reference framework for training massive transformer models at scale, supporting advanced parallelism strategies and mixed-precision training.