NVIDIA-NeMo/labs-molt

An agentic-first RL framework for research (9k lines).

What it solves

Molt is designed for high-velocity agentic RL research, specifically targeting the training of frontier-scale Mixture-of-Experts (MoE) models (up to 1T parameters). It addresses the complexity and infrastructure overhead typically associated with scaling RL from small models to massive ones, providing a PyTorch-native stack that remains hackable and readable while supporting fully asynchronous rollout and training.

How it works

Molt utilizes a three-component architecture coordinated by an asynchronous loop via Ray:

  1. Ray: Manages placement and asynchronous queues between components.
  2. vLLM: Handles the rollout (generation) phase.
  3. NVIDIA AutoModel + FSDP2: Manages the training phase in pure PyTorch, supporting advanced parallelism (TP, EP, CP) and Adam CPU offload for massive actors.

Rewards are defined in plain Python within an Env or ChatAgent class. The framework maintains a "token-first" contract, ensuring token IDs, logprobs, and multimodal tensors remain aligned from rollout through to training.

Who it’s for

It is built for AI researchers who need to iterate quickly on agentic environments and RL algorithms at scale, particularly those working with VLMs (Vision-Language Models) and large MoE models, without needing to rewrite their code when scaling from 8B to 1T parameters.

Highlights

  • Agentic-First Design: Uses a Gymnasium-aligned API where the agent is the program, allowing researchers to iterate on environments in Python without touching the trainer.
  • Frontier-Scale MoE: Native support for TP/EP/CP and MoE-native training, capable of scaling to 1T-class models like DeepSeek-V3.
  • Fully-Async Runtime: Overlaps rollout, training, and weight synchronization to keep large actors fed.
  • Small Codebase: Approximately 9.2K lines of RL code, making it easy to read and hack end-to-end.
  • Broad Algorithm Support: Includes REINFORCE, RLOO, GRPO, PPO (via GAE), and on-policy distillation.
  • IS Correction: Implements various importance sampling correction schemes (TIS, IcePop, MIS) to handle logprob mismatches between async rollout and training.

Related

  • Project
    PrimeIntellect-ai/prime-rl

    A framework for large-scale asynchronous reinforcement learning designed to train frontier models with 1T+ parameters on up to 1000+ GPUs.

  • Project
    meta-pytorch/torchforge

    A PyTorch-native agentic RL library that separates infrastructure concerns from model concerns to simplify scalable reinforcement learning experimentation.

  • Project
    alibaba/ROLL

    An efficient scaling library for reinforcement learning with LLMs that optimizes training and inference across large GPU clusters using a distributed Ray-based architecture.

  • Project
    areal-project/AReaL

    AReaL is an open‑source, fully asynchronous reinforcement‑learning framework for training large reasoning and agentic models. It offers modular micro‑services, supports many RL algorithms, runs on GPUs, NPUs or Ray clusters, and ships with dozens of ready‑to‑run examples (math, coding, search, vision‑language, RLHF, etc.).

  • Project
    HorizonRobotics/alf

    ALF (Agent Learning Framework) is a PyTorch‑based reinforcement‑learning library offering a large collection of algorithms (A2C, PPO, SAC, MuZero, etc.), modular components, Python‑style configuration, and support for single‑ and multi‑GPU/cluster training. It includes many ready‑to‑run examples (CartPole, Atari, MuJoCo, MetaDrive) and provides Docker, Nix, and uv‑based installation options.