NVIDIA-NeMo/Megatron-Bridge

Training library for Megatron-based models with bidirectional Hugging Face conversion capability

NVIDIA‑NeMo / Megatron‑Bridge – What It Is

NeMo Megatron Bridge is a PyTorch‑native library that lives inside NVIDIA’s NeMo framework. Its purpose is to act as a bridge between two major ecosystems for large‑scale models:

  1. 🤗 Hugging Face model format (the de‑facto standard for open‑source LLMs, vision‑language models, etc.)
  2. Megatron‑Core (the high‑performance, tensor‑ and pipeline‑parallel training engine originally from NVIDIA’s Megatron‑LM project).

The bridge lets you convert checkpoints back and forth, verify that the conversion is correct, and then train or fine‑tune the model with Megatron‑Core’s massive parallelism (tensor, pipeline, and mixed‑precision support). It also ships ready‑made training “recipes” for many state‑of‑the‑art models (Llama 3.2, GLM‑5, Qwen‑3, Nemotron‑3, Gemma‑4 VL, etc.).

Why It Matters

  • Speed & scale – Megatron‑Core can run trillion‑parameter models across dozens of GPUs with advanced features like CUDA‑graph execution, FP8/MXFP4 quantisation, and context‑parallelism. The bridge lets you bring any Hugging Face model into that world.
  • One‑stop workflow – Convert a public HF checkpoint → Megatron checkpoint → train/fine‑tune → export back to HF, all with the same Python API.
  • Broad model coverage – The repo tracks day‑0 support for dozens of recent LLMs, multimodal models, and diffusion models, and it is updated frequently (see the news section).
  • Community‑ready – Includes Docker containers, CI, code‑coverage badges, and a “skills” directory for AI‑coding assistants to help you add new models.

Core Features (as described in the README)

Category What the bridge does
Conversion Bidirectional checkpoint conversion between Hugging Face and Megatron formats; verification of weight parity.
Training PyTorch‑native training loop built on Megatron‑Core; supports pre‑training, supervised‑fine‑tuning (SFT), and LoRA/PEFT fine‑tuning.
Parallelism Tensor‑model‑parallel, pipeline‑model‑parallel, and context‑parallel execution; works on single‑GPU or multi‑GPU clusters.
Precision Mixed‑precision training/inference: BF16, FP8, FP4, MXFP4, etc.
Recipes Ready‑made config files for many models (e.g., llama32_1b_pretrain_1gpu_h100_bf16_config).
Multimodal Supports vision‑language, audio, and diffusion models (e.g., Gemma‑4 VL, Nemotron‑Diffusion).
PEFT LoRA and other parameter‑efficient fine‑tuning workflows are baked into the recipes.
Performance tooling Performance recipes, deterministic runs, CUDA‑graph optimisations, and integration with MLPerf‑style benchmarking.
Container support Official NeMo Framework Docker images provide the “best experience”.
Extensibility Submodule for Megatron‑Core (switchable between main and dev); skills/ directory for AI‑assistant guidance.

Typical Workflow (quick‑start style)

  1. Pick a model – e.g. meta-llama/Llama-3.2-1B from Hugging Face.
  2. from megatron.bridge import AutoBridge
bridge = AutoBridge.from_hf_pretrained("meta-llama/Llama-3.2-1B", trust_remote_code=True)
  3. Create a Megatron provider, set parallelism sizes, and materialise the model.
  4. Train – either run a supplied recipe (pretrain(cfg, forward_step)) or write your own training loop.
  5. Export – bridge.save_hf_pretrained(...) writes a full HF folder, or stream weights with export_hf_weights.

All of this can be launched with torch.distributed.run inside the NeMo container.

Who Should Use It?

  • Researchers who need to experiment with the latest LLMs at scale and want to leverage NVIDIA’s parallelism without rewriting the model.
  • Engineers building custom fine‑tuning pipelines (SFT, LoRA) that must run on multi‑GPU clusters.
  • ML‑ops teams that want a reproducible, containerised environment for large‑scale training and checkpoint conversion.

Installation & Getting Started

  • Preferred: Pull the official NeMo Framework Docker image (nvcr.io/nvidia/nemo:<TAG>) and run the container as shown in the README.
  • Development: Clone the repo, initialise the Megatron‑Core submodule, and use uv sync (or pip install -e .) to install the Python package.
  • Switching Megatron‑Core versions – scripts/switch_mcore.sh lets you toggle between the stable main commit and the experimental dev branch.

Current Model Coverage (selected)

  • LLMs: Llama 3.2, GLM‑5.2, Qwen 3.5/3.6, DeepSeek V4, Falcon H1, Gemma 4, Nemotron‑3 Super/Ultra/Nano Omni, K‑EXAONE‑2, MiniMax‑M2.5, Kimi‑K2.5 VL, etc.
  • Multimodal: Gemma 4 VL, Nemotron‑Diffusion, Qwen‑3 ASR, Bailing MoE V2.
  • Diffusion & generative vision models via the “Nemotron‑Labs Diffusion” collection.

License & Community

  • The repo is open‑source under the NVIDIA Software License (see LICENSE).
  • CI badges show test coverage and CI status; contributions are welcomed via the CONTRIBUTING.md guide.

Bottom line: NeMo Megatron Bridge is a genuine, actively maintained open‑source project that enables large‑scale training, fine‑tuning, and checkpoint conversion for modern LLMs and multimodal models using NVIDIA’s Megatron‑Core engine.

Related

  • Dispatch
    Training Language Models with Megatron-LM

    Hugging Face provides a guide on using NVIDIA's Megatron-LM framework to efficiently pre-train large language models on GPUs, including integration with the Transformers library.

  • Project
    NVIDIA-NeMo/Automodel

    A framework for automating the fine-tuning and pretraining of large-scale AI models with PyTorch-native parallelism and optimized recipes.

  • Dispatch
    NVIDIA NeMo AutoModel Accelerates Fine-Tuning of Mixture-of-Experts Models

    NVIDIA NeMo AutoModel provides 3.4-3.7x higher training throughput and 29-32% lower GPU memory usage for fine-tuning Mixture-of-Experts models by building on Hugging Face Transformers v5 with Expert Parallelism, DeepEP, and TransformerEngine kernels, requiring only a one-line import change.

  • Project
    EleutherAI/gpt-neox

    A library for training large-scale autoregressive language models on GPUs, providing distributed training capabilities and optimizations for models with billions of parameters.

  • Project
    NVIDIA-NeMo/RL

    A scalable post-training library for multimodal models that streamlines reinforcement learning and alignment using high-performance backends like Megatron Core and vLLM.