haoyi-duan/WorldScore

📚 What is WorldScore?

WorldScore is an open‑source benchmark for evaluating world‑generation models – systems that create 3‑D scenes, 4‑D spatio‑temporal worlds, or videos from text or images. It provides:

• A curated dataset (WorldScore‑Dataset) of initial scenes and camera‑movement instructions.
• Reference implementations for downloading the data, generating videos with any model, and running a suite of spatial‑temporal metrics.
• A public leaderboard (hosted on HuggingFace) where researchers can submit their scores.

The project accompanies the ICCV‑2025 paper “WorldScore: A Unified Evaluation Benchmark for World Generation” and aims to give a single number that reflects how well a model can extend a world and follow camera paths – something existing video‑quality benchmarks miss.

---

🚀 Getting Started (high‑level steps)

1. Clone & configure – set WORLDSCORE_PATH, MODEL_PATH, and DATA_PATH in a .env file.
2. Install – create a conda env (Python 3.10) and run pip install -e . plus a long list of dependencies (PyTorch 2.5, DROID‑SLAM, Grounding‑SAM, SAM‑2, VFIMamba, etc.).
3. Download the dataset – python download.py pulls the WorldScore‑Dataset from HuggingFace.
4. Add your model –
   • Write a tiny YAML config (model_name.yaml) describing resolution, generation type (image‑to‑video i2v or text‑to‑video t2v), frame count, fps, etc.
   • Register the model name in worldscore/benchmark/utils/modeltype.py under the appropriate family (threedgen, fourdgen, videogen).
   • Implement a Python class in world_generators/ exposing a generate_video(prompt, image_path=None) method that returns a list of PIL.Image objects or a [N,3,H,W] tensor.
5. Generate videos – run python world_generators/generate_videos.py --model-name <your_model> (single‑GPU) or use the Slurm wrapper for multi‑GPU clusters.
6. Run evaluation – after generation, execute python worldscore/run_evaluate.py --model_name <your_model> (or the Slurm variant). The script computes a collection of metrics (e.g., depth consistency, object tracking, grounding quality) using the pre‑downloaded checkpoints for DROID‑SLAM, Grounding‑DINO, SAM, etc.
7. Submit results – a JSON file worldscore_output/worldscore.json is produced. Email it to haoyiduan@princeton.edu or upload videos for the team to evaluate; the leaderboard updates automatically.

---

📊 What the benchmark measures

WorldScore combines spatial and temporal criteria:

• Geometric consistency – via DROID‑SLAM pose estimation.
• Object grounding – using Grounding‑DINO and Segment‑Anything (SAM / SAM‑2).
• Temporal coherence – assessing motion smoothness and continuity.
• Prompt adherence – checking whether the generated world follows the supplied camera path and scene‑expansion instructions.

The exact metric list lives in worldscore/benchmark/metrics/ and is extensible; new models can plug into the same pipeline without rewriting evaluation code.

---

🏅 Leaderboard & Community

Results are displayed on a HuggingFace Space. Teams can either:

• Self‑evaluate and submit their JSON scores (recommended – instant update), or
• Send raw videos to the authors for manual scoring.

The repo also hosts example adapters for popular world‑generation models such as WonderJourney, WonderWorld, CogVideoX, DynamiCrafter, and many 3‑D/4‑D systems, showing how to integrate them.

---

📦 Quick reference (key files)

Path	Purpose
download.py	Pulls the WorldScore dataset from HuggingFace.
world_generators/	Model adapters and video‑generation scripts.
worldscore/run_evaluate.py	Main entry‑point for metric computation.
worldscore/benchmark/metrics/	Implementations of the individual evaluation metrics.
config/model_configs/	YAML templates for new models.
worldscore/benchmark/utils/modeltype.py	Registry mapping model families to names.

---

📖 Citation

@article{duan2025worldscore,
  title={WorldScore: A Unified Evaluation Benchmark for World Generation},
  author={Duan, Haoyi and Yu, Hong-Xing and Chen, Sirui and Fei-Fei, Li and Wu, Jiajun},
  journal={arXiv preprint arXiv:2504.00983},
  year={2025}
}

---

In short: WorldScore gives researchers a ready‑to‑run, reproducible way to test how well their generative models can build and navigate virtual worlds, filling a gap left by traditional video‑quality benchmarks.