NVIDIA Cosmos 3 Open Models for Interactive World Simulation

Overview of Cosmos 3 Open Models

NVIDIA has expanded its Cosmos world model platform by releasing four open-weight checkpoints within the Cosmos 3 family. These 16B parameter models are designed to facilitate interactive world simulation and multi-sensor, multi-view driving scene generation. Released under the OpenMDW1.1 license, these models provide a foundation for researchers and developers to build, adapt, and refine simulated environments for autonomous systems and robotics.

Model Capabilities

The four new checkpoints serve distinct roles in simulation and data generation:

  • Cosmos3-Nano-Transfer-Auto: A bidirectional model for offline synthetic driving data generation. It supports synchronized video across up to 11 camera views and joint RGB–LiDAR generation, guided by World Scenario Map (WSM) controls and text prompts.
  • Cosmos3-Nano-Sim-Auto: A causally distilled streaming variant of Transfer-Auto. It generates successive chunks of synchronized camera video and LiDAR in few-step denoising, enabling long, interactive driving rollouts.
  • Cosmos3-Nano-Sim-Transfer: Designed to convert physics simulator outputs—such as depth, segmentation, edges, and blur—into photorealistic RGB video, allowing simulators to maintain ground-truth state while generating visual observations.
  • Cosmos3-Nano-Sim-Bimanual: Focused on robotics, this model predicts visual consequences based on bimanual robot actions, supporting interactive simulation for embodiments like YAM and AgiBot.

Technical Architecture and Training

Cosmos 3 utilizes a Mixture-of-Transformers backbone that integrates separate text-processing and visual-generation networks. The system operates on compressed video latents, where temporal-causal attention allows the model to process history and current controls while preventing leakage from future frames.

The training process follows a four-stage recipe designed to improve stability and performance:

  1. Bidirectional Adaptation: Learning domain-specific control-to-video relationships from recorded clips.
  2. Teacher Forcing: Predicting chunks from historical observations while masking future targets.
  3. Consistency Distillation: Training a student model to generate chunks in four denoising steps using a frozen teacher.
  4. Self-Forcing: Training on the student's own generated history to mitigate error compounding, where a frozen teacher and score model provide distribution-matching signals.

Implementation and Adaptation

NVIDIA provides post-training recipes for each model, allowing developers to adapt the checkpoints to specific sensor rigs, robot embodiments, or driving domains. The framework requires a Linux environment with an NVIDIA GPU and the NVIDIA Container Toolkit. Developers can use the provided inference skills and notebooks to integrate these models into existing simulation pipelines. The models are intended to serve as a starting point for researchers to measure how well synthetic environments support policy learning and to contribute improvements to the underlying simulation technology.

Sources