NVIDIA-NeMo/Automodel
🚀 Pytorch Distributed native training library for LLMs/VLMs with OOTB Hugging Face support
What it solves
NeMo AutoModel simplifies the process of pretraining and fine-tuning large-scale AI models. It removes the manual effort of adapting models from platforms like Hugging Face to high-performance training environments by automating the integration and providing optimized training recipes.
How it works
The project provides a framework that integrates Hugging Face models and applies PyTorch-native parallelism (such as FSDP2, expert parallelism, pipeline parallelism, and context parallelism) and custom kernels to optimize performance. It uses a Pytorch DTensor-native SPMD library to handle large-scale training across multiple GPUs and nodes.
Who it’s for
It is designed for AI researchers and developers who need to train or fine-tune massive models (including LLMs, VLMs, and diffusion models) using high-performance hardware like NVIDIA H100 or GB200 GPUs.
Highlights
- Broad Model Support: Supports a wide array of architectures including MoE (Mixture-of-Experts), dense models, and multimodal decoders.
- Optimized Parallelism: Implements advanced parallelism strategies like Expert Parallelism (EP), Pipeline Parallelism (PP), and Context Parallelism (CP).
- Speculative Decoding: Includes tools to train target-aligned drafters (e.g., EAGLE, DFlash, DSpark) to speed up inference.
- Ready-to-Use Recipes: Provides pre-configured YAML recipes for specific models and datasets (e.g., HellaSwag, MedPix) to accelerate deployment.
Related
- Project
- Dispatch
- Project
- Dispatch
- Project