NVIDIA/cosmos

What it solves

NVIDIA Cosmos provides a foundation for building "Physical AI" applications, such as robots, autonomous vehicles, and smart infrastructure. It addresses the need for models that can not only understand the world but also simulate it, predict future states, and generate actionable policies for physical movement in the real world.

How it works

Cosmos 3 uses a unified Mixture-of-Transformers architecture to create omnimodal world models. It is split into two primary functional surfaces:

• Reasoner: Takes text and vision as input to produce text, enabling world understanding, grounding, task planning, and embodied reasoning.
• Generator: Takes text, vision, sound, and action as input to produce vision, sound, and action, enabling world simulation, synthetic data generation, and policy learning.

Who it’s for

It is designed for developers of physical AI, including those working on robotics, autonomous driving, and synthetic data generation for training physical agents.

Highlights

• Three Model Tiers: Offers Super (64B) for high-quality synthetic data, Nano (16B) for balanced speed/quality, and Edge (4B) for real-time on-device deployment.
• Omnimodal Capabilities: Handles a wide array of inputs and outputs including text, vision, sound, and physical actions.
• Physics-Aware Generation: Capable of rendering momentum transfer and physical fidelity (e.g., Newton's cradle).
• Action-Conditioned World Modeling: Supports forward dynamics, such as egocentric rollouts from camera and hand pose inputs.