nvidia-cosmos/cosmos-predict1

What it solves

Cosmos-Predict1 provides world foundation models (WFMs) designed for future state prediction. It allows users to generate visual simulations of future world states based on text or video prompts, which is essential for developing Physical AI and embodied agents.

How it works

The project utilizes two primary model architectures to predict future visual states:

• Diffusion-based models: Used for Text2World and Video2World generation, as well as world interpolation and creating multi-view clips from a single video.
• Autoregressive-based models: Used for Video2World generation, taking video prompts and optional text prompts to simulate future states.

To support these models, the project includes specialized image and video tokenizers that compress visual data into continuous latent vectors or discrete integers. It also provides post-training scripts so developers can adapt pre-trained models to specific applications.

Who it’s for

It is designed for Physical AI builders and researchers who need to simulate environments, predict future visual outcomes, or create synthetic training data for embodied agents.

Highlights

• Multi-modal generation: Supports Text2World and Video2World workflows.
• Diverse model options: Offers both diffusion and autoregressive architectures in various sizes (from 4B to 14B parameters).
• Advanced visual tools: Includes a WorldInterpolator for higher-FPS video and Single2MultiView capabilities for dynamic multi-view clips.
• Efficient Tokenization: Provides a suite of continuous and discrete tokenizers for images and videos at different resolutions.