google-deepmind/physics-IQ-benchmark

Physics‑IQ & Physics‑IQ Verified – a benchmark for generative video models

What it is – A high‑quality dataset of real‑world videos (4K @ 30 fps) that captures a wide variety of physical phenomena (collisions, fluids, gravity, magnetism, light, etc.) from three camera angles and with two recordings per scenario. The repo provides the evaluation workflow for two versions of the benchmark:

• Physics‑IQ – the original release.
• Physics‑IQ Verified – an improved variant with better prompts, masks and scoring (the recommended one).

Why it matters – Generative video models (text‑to‑video, video‑to‑video, etc.) are increasingly powerful, but there is no standard way to test whether the videos they produce obey the laws of physics. This benchmark supplies a realistic test set and a reproducible scoring pipeline, enabling fair comparison across models.

How to use it

1. Download the data from the Hugging Face hub (Anates‑Labs‑Research/Physics‑IQ‑Verified). The dataset contains:
   • full‑videos/ – the original 5‑second clips.
   • split‑videos/testing/ – the same clips split into conditioning and target parts for V2V models.
   • switch‑frames/ – single‑frame images used as seeds for I2V models.
   • video‑masks/real/ – ground‑truth masks needed for the scoring metric.
2. Set up the environment – either with the modern uv tool (uv sync) or with pip (pip install .). Linux + ffprobe is required.
3. Pick a prompt template – the benchmark supports two prompt styles:
   • bpp (best‑practice‑prompt) – model‑specific prompts generated by a templater in descriptions/best_practice/.
   • op (original prompt) – the raw prompts from the original dataset. You can generate a new CSV for a custom model via uv run physiq/generate_descriptions.py <model_name>.
4. Generate videos –
   • I2V models: feed the single frames from switch‑frames/ plus the chosen description CSV.
   • V2V models: feed the conditioning videos from split‑videos/conditioning‑videos/ (and optionally the description CSV). Save each output video with the benchmark ID prefix (0001_… .mp4).
5. Organise runs – create a folder per model‑run named <model>-<prompt_setting>-run_<01‑04>. Four independent runs are recommended for a statistically sound leaderboard score.
6. Trim to 5 s – ensure every generated clip is exactly five seconds long before evaluation.
7. Run the scorer – execute physiq/run_physics_iq.py (or the equivalent entry point). The script compares your videos against the ground‑truth masks and returns a percentage score (100 % = perfect physical realism).
8. Submit to the leaderboard – add a row to the markdown table via a pull request if you want your result displayed.

Leaderboard – The README lists current scores (e.g., Magi‑1 24B + GeoPhys ≈ 58 % on the Verified benchmark, Cosmos3‑Super ≈ 39 %). Scores are reported as mean ± standard‑deviation over the four runs.

Citation & licensing – The project is released under a permissive license (see the License section) and asks users to cite the accompanying arXiv report when publishing results.

---

Bottom line – Physics‑IQ Verified gives researchers a ready‑to‑use, real‑world video suite and a reproducible evaluation script to measure how well generative video models understand and respect physical laws. It is a genuine, community‑oriented benchmark in the AI‑generated video space.