nv-tlabs/omni-dreams

NVIDIA OmniDreams is a world model that generates photorealistic video for autonomous-driving simulation in real time.

What it solves

OmniDreams is a world model designed for autonomous driving simulation. It solves the problem of generating photorealistic, multi-camera video in real time, allowing for the creation of synthetic driving environments that can be used to train and test autonomous vehicle systems.

How it works

The model uses an autoregressive generation process. It takes a single real RGB frame as an anchor for scene appearance, a text prompt for driving context, and per-frame HD map images and trajectory poses for structured conditioning. It generates video frames in chunks, feeding each chunk back into the next step to maintain temporal consistency over time.

Who it’s for

This project is primarily for researchers and developers working on autonomous driving simulation and world models, specifically those with access to high-end NVIDIA hardware (e.g., 8-GPU Ampere/Hopper nodes).

Highlights

  • Real-time generation: Produces photorealistic multi-camera video for simulation.
  • Structured conditioning: Uses HD maps and trajectory poses to guide the generation.
  • Post-training capabilities: Includes tools for fine-tuning the Cosmos2 SV-HDMap world model using various distillation methods.
  • Integration: Works alongside the companion project FlashDreams for interactive driving and batch inference.

Related

  • Dispatch
    NVIDIA Cosmos-H-Dreams: Real-Time Generative Simulation for Surgical Robotics

    NVIDIA has introduced Cosmos-H-Dreams, a real-time, action-conditioned generative simulator that distills a surgical world model into a causal student model to enable interactive surgical robotics simulation at 160 FPS.

  • Project
    nvidia-cosmos/cosmos-predict1

    A suite of world foundation models for future state prediction that generates visual simulations from text or video prompts to support Physical AI development.

  • Project
    NVIDIA/cosmos

    NVIDIA Cosmos is a platform of omnimodal world models designed for physical AI, enabling robots and autonomous vehicles to reason about, simulate, and act within the physical world.

  • Project
    AMAP-ML/DreamX-World

    DreamX-World is a general-purpose interactive world model that generates high-fidelity, controllable simulations allowing users to explore and transform environments via event prompts.

  • Project
    nvidia-cosmos/cosmos-transfer2.5

    A multi-controlnet platform for Physical AI that transforms video modalities (like depth and segmentation) into high-fidelity training data for robots and autonomous vehicles.