NVIDIA-NeMo/RL

Scalable toolkit for efficient model reinforcement

🎯 What is NeMo RL?

NeMo RL is NVIDIA’s open‑source post‑training reinforcement‑learning library built on top of the larger NeMo framework. It lets you take large multimodal models (LLMs, vision‑language models, MoE models, etc.) and fine‑tune them with RL‑based objectives such as GRPO, DPO, GDPO, DAPO, or on‑policy distillation. The focus is on scalability – you can run a quick single‑GPU experiment or launch massive multi‑node, multi‑GPU jobs that use advanced parallelism (Tensor, Pipeline, Expert, FSDP2, etc.) via either PyTorch DTensor or NVIDIA’s Megatron Core.

🚀 Core Capabilities (as of the README)

Capability Details
Algorithms GRPO/GSPO/DAPO, GDPO, LoRA‑enhanced RL (GRPO, DPO), supervised fine‑tuning (SFT), on‑policy distillation, multi‑turn RL, X‑Token off‑policy distillation.
Model Support Major LLM families (Nemotron‑3, Qwen 3.5, Gemma 4, GLM 5.1, Minimax‑M3, etc.), both dense and MoE variants, plus vision‑language models.
Training Backends • DTensor – PyTorch‑native distributed training (TP, SP, PP, CP, FSDP2).
• Megatron Core – NVIDIA’s 6‑D parallelism engine for the biggest models.
Generation / Rollout Backends vLLM (high‑throughput inference) and Megatron‑Native inference (no weight conversion).
Performance Features Muon optimizer, speculative decoding, FP8 low‑precision training, sequence packing, long‑context (YARN) support, async rollouts/replay buffers, Ray‑based resource orchestration, container images for GB200 GPUs.
Integration Hugging Face model loading, NeMo‑Gym environment support, seamless Docker/NGC container deployment, extensive YAML‑based recipe system.
Documentation & tooling Full docs site, design docs, example configs, Google‑Colab notebooks, CI‑tested releases, community discussions.

📚 Typical Use‑Cases

Scenario How NeMo RL helps
Aligning a large LLM (e.g., Nemotron‑3‑Ultra) with human preferences via RLHF‑style training Use the GRPO/DAPO recipes, pick a backend (DTensor for moderate size, Megatron for >100 B), and run on a multi‑node GPU cluster or the provided NGC container.
Improving a vision‑language model with reward‑based fine‑tuning The library supports VLMs and provides GRPO recipes that work with image‑text datasets, leveraging the same parallelism infrastructure.
Research on new RL algorithms (e.g., GDPO, X‑Token distillation) All algorithms are exposed as modular components; you can swap them in a YAML config and run a quick smoke test (grpo_smoke.yaml).
Fast prototyping on a laptop or single GPU The native PyTorch (DTensor) path runs without Megatron, and the Docker quick‑start gives you a ready‑to‑run environment with vLLM and SGLang already installed.
Production‑scale RL serving Ray‑based distributed training + async rollouts + container images let you scale from research to production pipelines.

🛠️ Getting Started (quick recap)

  1. Pull the official container (recommended):
    docker pull nvcr.io/nvidia/nemo-rl:latest
  2. Clone the repo (includes submodules):
    git clone --recursive https://github.com/NVIDIA-NeMo/RL.git nemo-rl
cd nemo-rl
  3. Run a test job (choose backend):
    • DTensor: uv run python examples/run_grpo.py
    • Megatron: uv run examples/run_grpo.py --config examples/configs/grpo_math_1B_megatron.yaml
  4. Scale up by editing the provided YAML recipes (GPU count, parallelism flags) and launching with Ray or via the NGC container on a multi‑node cluster.

📌 Why It Matters

NeMo RL bundles together the latest RL‑based alignment algorithms, state‑of‑the‑art large‑model parallelism, and production‑ready tooling (Docker, Ray, vLLM). For anyone who wants to fine‑tune or align massive LLMs/VLMs without building the distributed stack from scratch, NeMo RL offers a single, well‑documented entry point.

TL;DR – NeMo RL is a genuine, actively maintained NVIDIA library for scaling reinforcement‑learning post‑training of large multimodal models, supporting many modern algorithms, backends, and hardware configurations.

Related

  • Project
    NVIDIA-NeMo/Automodel

    A library for automating the pretraining and fine-tuning of Hugging Face models using PyTorch-native parallelism and optimized recipes for large-scale NVIDIA hardware.

  • Dispatch
    Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

    Hugging Face surveyed 16 open-source RL libraries and found that async RL training separates inference and training onto different GPU pools, uses a rollout buffer, and pushes weights asynchronously, with Ray dominating orchestration and NCCL broadcast the common weight‑sync method.

  • Dispatch
    NVIDIA NeMo AutoModel Accelerates Fine-Tuning of Mixture-of-Experts Models

    NVIDIA NeMo AutoModel provides 3.4-3.7x higher training throughput and 29-32% lower GPU memory usage for fine-tuning Mixture-of-Experts models by building on Hugging Face Transformers v5 with Expert Parallelism, DeepEP, and TransformerEngine kernels, requiring only a one-line import change.

  • Project
    huggingface/trl

    TRL is a Hugging Face library that provides ready‑made trainer classes for post‑training large language models using modern alignment methods (SFT, GRPO, DPO, KTO, reward modeling, distillation). It integrates with Transformers, Accelerate, PEFT, DeepSpeed and offers a simple CLI.

  • Project
    alibaba/ROLL

    An efficient scaling library for reinforcement learning with LLMs that optimizes training and inference across large GPU clusters using a distributed Ray-based architecture.