nvidia-cosmos/cosmos-predict2.5

NVIDIA Cosmos Predict 2.5 – Video‑based World‑Simulation Model

What it is – Cosmos‑Predict 2.5 is an open‑source world‑foundation model (WFM) from NVIDIA’s Cosmos platform. It is a diffusion‑style generative model that takes text, images or video as input and produces future‑state video predictions. The model is built for Physical AI: robotics, autonomous‑vehicle simulation, video‑analytics, and any system that needs a physically plausible forecast of how a scene will evolve.

Why it matters – It unifies three tasks that were previously separate:

• Text‑to‑World – generate a video from a textual description.
• Image‑to‑World – animate a static picture into a plausible video.
• Video‑to‑World – continue or transform an input video into a future‑state video. All three share the same backbone and the same Cosmos‑Reason 1 vision‑language encoder, giving consistent quality and better alignment with prompts.

---

Key Features (as described in the README)

Feature	Detail
Model sizes	2 B‑parameter and 14 B‑parameter checkpoints, both pre‑trained and post‑trained variants.
Multimodal inputs	Accepts text + image, text + video, or pure text (distilled version).
Domain‑specific variants	• auto/multiview – tuned for autonomous‑vehicle 7‑camera rigs. • robot/action‑cond – action‑conditioned generation for robotic manipulation. • robot/multiview‑agibot – 3‑camera robot data (AgiBot). • robot/policy – policy‑conditioned model trained on Libero & RoboCasa.
Post‑training recipes	LoRA fine‑tuning, DMD2 distillation, DreamGen dataset, gr00t‑dreams dataset, and a full cookbook of scripts for custom adaptation.
Inference flexibility	Native PyTorch pipelines, Diffusers integration, Blackwell + ARM GPU support, guard‑rail off‑loading, and multi‑storage asset handling.
Accelerated tokenization	Uses CUDA‑accelerated tokenizers and torch.compile for faster inference.
Guardrails	Built‑in safety checks to keep generated videos physically plausible and within policy limits.
Documentation & examples	Detailed setup guide, troubleshooting, Jupyter notebooks, and a dedicated Cosmos‑Cookbook with step‑by‑step recipes.

---

Typical Use‑Cases

• Robotics – generate realistic video of a robot arm performing a task, or condition generation on a planned action sequence to evaluate outcomes before execution.
• Autonomous driving – simulate future traffic scenes from multi‑camera inputs for scenario testing and data augmentation.
• Video analytics – predict how a surveillance scene will evolve (e.g., crowd movement) to improve proactive decision‑making.
• Content creation – turn a storyboard or a single image into a short, physically coherent video clip.
• Model research – serve as a baseline for studying multimodal diffusion, rectified‑flow training, and physical reasoning in generative models.

---

Getting Started (quick‑start steps distilled from the README)

1. Clone the repo

   git clone https://github.com/nvidia-cosmos/cosmos-predict2.5.git
   cd cosmos-predict2.5
   

2. Install dependencies (the repo provides a docs/setup.md with a conda/Docker recipe; the simplest is the provided Docker image).

   # Example using conda
   conda create -n cosmos-predict python=3.10
   conda activate cosmos-predict
   pip install -r requirements.txt
   

3. Download a model checkpoint – pick a size from Hugging Face:

   huggingface-cli download nvidia/Cosmos-Predict2.5-2B --repo-type model --local-dir ./models/2b/base
   

4. Run a basic inference – generate a video from a text prompt:

   from cosmos_predict import CosmosPredictPipeline
   pipe = CosmosPredictPipeline.from_pretrained("./models/2b/base")
   video = pipe("A night‑time city bus terminal slowly comes to life, buses start moving.")
   video.save("output.mp4")
   

5. Explore the examples – the examples/notebook/ folder contains ready‑to‑run Jupyter notebooks for Image2World, Video2World, robot‑action‑conditioned generation, and multi‑view autonomous‑vehicle scenarios.

---

Community & Contributions

• Cosmos‑Cookbook – a companion repo (nvidia-cosmos/cosmos-cookbook) with ready‑made recipes for fine‑tuning, distillation, and deployment.
• Issues & PRs – the project welcomes contributions; see CONTRIBUTING.md for the workflow.
• Future direction – development has shifted to Cosmos 3, a unified model that adds reasoning, policy generation, and cross‑modal transfer. Existing users are encouraged to migrate, but Cosmos‑Predict 2.5 will continue to receive limited maintenance.

---

License

• Code – Apache 2.0.
• Model weights – NVIDIA Open Model License (commercial‑friendly, with a separate custom‑license option via cosmos-license@nvidia.com).

---

TL;DR

Cosmos‑Predict 2.5 is NVIDIA’s open‑source, diffusion‑based video‑generation model for predicting future world states from text, images, or video. It ships in 2 B and 14 B sizes, includes domain‑specific variants for robotics and autonomous driving, and comes with a full suite of fine‑tuning and deployment tools. Ideal for researchers and engineers building physical‑AI systems that need realistic, controllable video simulations.