nvidia-cosmos/cosmos-rl

Cosmos-RL is a flexible and scalable Reinforcement Learning framework specialized for Physical AI applications.

What it solves

Cosmos-RL is designed to handle the complexities of large-scale Reinforcement Learning (RL) training for Physical AI applications, providing a scalable and flexible framework to manage the heavy computational demands of training embodied agents.

How it works

The framework uses a decoupled, single-controller architecture that separates the training process into specialized roles:

  • Policy (Consumer): Dedicated replicas for training instances.
  • Rollout (Producer): Dedicated replicas for generation engines.
  • Controller: An efficient messaging system that coordinates weights and rollouts between the policy and rollout replicas.

To maximize performance, it supports multiple parallelism strategies (Tensor, Sequence, Context, FSDP, and Pipeline) and utilizes low-precision training and rollout (FP8 and FP4) to reduce memory and compute overhead.

Who it’s for

It is intended for developers and researchers working on Physical AI, such as robotics, autonomous vehicles, and smart spaces, who need to reach large-scale RL training workloads.

Highlights

  • Advanced Parallelism: Supports five different types of parallelism to distribute workloads across GPUs.
  • Asynchronous Architecture: Decouples policy and rollout replicas for better efficiency.
  • Elastic Training: Uses dynamic NCCL Process Groups to allow GPUs to be registered or unregistered on-the-fly, ensuring fault tolerance.
  • Low-Precision Support: Optimized for FP8 and FP8/FP4 precision to speed up training and rollouts.

Related

  • Project
    PrimeIntellect-ai/prime-rl

    A framework for large-scale asynchronous reinforcement learning designed to train 1T+ parameter MoE models across 1000+ GPUs using FSDP2 and vLLM.

  • Project
    radixark/miles

    An enterprise-grade reinforcement learning framework for large-scale model post-training that integrates SGLang for rollouts and Megatron-LM for scalable training.

  • Project
    RLinf/RLinf

    A flexible and scalable reinforcement learning infrastructure designed to streamline the training of Embodied and Agentic AI across diverse hardware and simulators.

  • Project
    alibaba/ROLL

    An efficient scaling library for reinforcement learning with LLMs that optimizes training and inference across large GPU clusters using a distributed Ray-based architecture.

  • Project
    ServiceNow/PipelineRL

    A scalable asynchronous reinforcement learning framework for LLM agents that uses in-flight weight updates to maximize GPU utilization while maintaining on-policy data freshness.