NVIDIA-NeMo/RL

What it solves

NeMo RL provides a scalable and efficient framework for the post-training of multimodal models, including Large Language Models (LLMs) and Vision Language Models (VLMs). It streamlines the process of applying reinforcement learning (RL) and alignment techniques to move models from pre-training to production-ready performance, supporting both small-scale research experiments and massive multi-node deployments.

How it works

The library uses a modular architecture that separates training and generation (rollout) processes:

• Training Backends: It supports DTensor (PyTorch-native distributed training) for memory efficiency and Megatron Core for high-performance scaling to very large models using 6D parallelism.
• Generation Backends: It integrates with vLLM and SGLang for high-throughput inference, as well as a Megatron native backend to eliminate weight conversion between training and inference.
• Infrastructure: It leverages Ray for scalable resource management and process isolation between RL actors.
• Optimization: It supports advanced techniques like sequence packing, FP8 low-precision training, and speculative decoding to accelerate the RL loop.

Who it’s for

NeMo RL is designed for AI researchers and engineers who need to post-train large-scale multimodal models using RL, specifically those working with massive datasets and hardware clusters (such as GB200) where efficiency and scalability are critical.

Highlights

• Diverse RL Algorithms: Supports GRPO, GSPO, DAPO, DPO, SFT (with LoRA), and on-policy/off-policy distillation.
• Broad Model Support: Compatible with a wide range of models including Qwen, Nemotron, Gemma 4, and GLM.
• Advanced Parallelism: Implements TP, PP, CP, SP, EP, and FSDP2 for handling massive models and long context lengths.
• Multimodal Capabilities: Supports SFT and GRPO specifically for Vision Language Models (VLMs).
• Seamless Integration: Provides out-of-the-box support for Hugging Face models and integration with NeMo-Gym.