NVIDIA-NeMo/Megatron-Bridge

What it solves

NeMo Megatron Bridge solves the difficulty of moving models between the Hugging Face ecosystem and NVIDIA's high-performance Megatron Core. It provides a standardized way to convert checkpoints bidirectionally, verify their integrity, and leverage Megatron Core's advanced parallelism (tensor and pipeline) for large-scale training and fine-tuning without having to rewrite model definitions.

How it works

The library acts as a conversion and verification layer. It allows users to import Hugging Face models, convert them into a format compatible with Megatron Core, and then use a PyTorch-native training loop to perform pretraining, Supervised Fine-Tuning (SFT), or Parameter-Efficient Fine-Tuning (PEFT/LoRA). It also supports exporting Megatron checkpoints back to Hugging Face format for use in various inference engines.

Who it’s for

It is designed for ML engineers and researchers who need to train or fine-tune massive models (LLMs, VLMs, and multimodal models) using NVIDIA hardware, specifically those who want the scalability of Megatron Core while maintaining compatibility with the Hugging Face Hub.

Highlights

• Bidirectional Conversion: Seamlessly move weights between Hugging Face and Megatron Core formats.
• High-Performance Training: Supports tensor and pipeline parallelism with mixed precision (FP8, BF16, FP4).
• Broad Model Support: Compatible with a wide range of architectures including Llama, DeepSeek, Qwen, Gemma, and Nemotron.
• Multimodal Capabilities: Supports vision-language (VLM), audio, and diffusion models.
• Verification Tools: Built-in mechanisms to ensure conversion accuracy and checkpoint integrity.