nv-tlabs/omni-dreams

NVIDIA Cosmos-Dreams (fka NVIDIA OmniDreams) is a world model that generates photorealistic video for autonomous-driving simulation in real time.

What it solves

Cosmos-Dreams is a world model designed for autonomous driving simulation. It solves the problem of generating photorealistic, multi-camera video in real time, allowing developers to simulate driving scenarios based on specific constraints and conditions.

How it works

The model uses an autoregressive process to generate video chunks. It takes a single real RGB frame as an anchor for scene appearance, a text prompt for driving context, and per-frame HD map images and trajectory poses to provide structured conditioning. Each generated chunk is then fed back into the model to continue the scene over time.

Who it’s for

This project is primarily for researchers and developers working on autonomous driving simulation and world models, specifically those with access to high-end NVIDIA hardware (such as 8-GPU Ampere/Hopper nodes).

Highlights

  • Real-time generation: Produces photorealistic multi-camera video for simulation.
  • Structured conditioning: Uses HD maps and trajectory poses to guide the video generation.
  • Post-training support: Includes launchers and configurations for fine-tuning the model using techniques like student-init, bidirectional teacher, and self-forcing distillation.
  • Integration: Works with the companion project FlashDreams for interactive driving and batch video generation.

Related

  • Dispatch
    NVIDIA Cosmos-H-Dreams: Real-Time Generative Simulation for Surgical Robotics

    NVIDIA has introduced Cosmos-H-Dreams, a real-time, action-conditioned generative simulator that distills a surgical world model into a causal student model to enable interactive surgical robotics simulation at 160 FPS.

  • Project
    NVIDIA/flashdreams

    A high-performance inference and serving library for interactive autoregressive video and world models, enabling real-time simulation for robotics, autonomous vehicles, and gaming.

  • Project
    nvidia-cosmos/cosmos-predict1

    A suite of world foundation models for future state prediction that generates visual simulations from text or video prompts to support Physical AI development.

  • Project
    NVIDIA/cosmos

    An open platform of omnimodal world models and tools for building Physical AI, enabling the joint processing and generation of text, vision, audio, and action sequences for robotics and autonomous systems.

  • Project
    AMAP-ML/DreamX-World

    DreamX-World is a general-purpose interactive world model that generates high-fidelity, controllable simulations allowing users to explore and transform environments via event prompts.