NVIDIA/cosmos

What it solves

NVIDIA Cosmos is an open platform designed to build "Physical AI" for robotics, autonomous vehicles, and smart infrastructure. It provides a unified framework for world models that can both understand and generate the physical world, bridging the gap between vision-language models, video generators, and world-action models.

How it works

Cosmos 3 uses a Mixture-of-Transformers (MoT) architecture that combines an autoregressive transformer for reasoning and a diffusion transformer for multimodal generation. It operates in two primary modes:

• Reasoner Mode: Processes text and vision inputs to produce text outputs for tasks like task planning, physical reasoning, and action forecasting.
• Generator Mode: Processes text, vision, sound, and action inputs to produce images, videos, synchronized sound, and action-conditioned rollouts.

The system uses a unified 3D multi-dimensional rotary position embedding (mRoPE) to maintain spatial and temporal consistency across different modalities.

Who it’s for

• Robotics Developers: For creating real-time robotic policies and action trajectories.
• AI Researchers: For developing world simulators and synthetic data generation.
• Autonomous System Engineers: For building decision-making systems for vehicles and smart infrastructure.

Highlights

• Omnimodal Capabilities: Jointly processes and generates language, images, video, audio, and action sequences.
• Flexible Model Family: Includes Super (64B), Nano (16B), and Edge (4B) models to suit different hardware from data centers to Jetson edge devices.
• Diverse Workflows: Supports text-to-video, image-to-video, video-to-video, and forward dynamics (predicting future states from actions).
• Broad Integration: Compatible with Diffusers, vLLM-Omni, TensorRT-LLM, and SGLang for both research and production deployment.