Vchitect/VBench

VBench – A Benchmark Suite for Video Generative Models

What is it? VBench (and its extensions VBench++ and VBench‑2.0) is an open‑source evaluation framework for modern video‑generation AI. It provides a unified code base that lets researchers and developers automatically score generated videos on a large set of well‑defined quality dimensions.

Why it matters Video‑to‑text models have exploded in capability, but there is no single, agreed‑upon way to measure how good the output is. VBench tackles this by breaking “video generation quality” into a hierarchy of evaluation dimensions (e.g., subject consistency, temporal flickering, motion smoothness, aesthetic quality, trustworthiness, intrinsic faithfulness, etc.) and supplying:

• Prompt suites – curated text prompts for each dimension and content category.
• Evaluation method suite – automatic metrics and pipelines that compute a score for each dimension.
• Human preference annotations – a validation layer showing that the automatic scores align with what people perceive as good.
• Leaderboards & arenas – public Hugging Face spaces where researchers can upload results, view sampled videos, and compare models.

Key components

Component	What it covers	Where it lives in the repo
VBench (original)	16 text‑to‑video dimensions (subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic quality, imaging quality, object class, multiple objects, human action, color, spatial relationship, scene, temporal style, appearance style, overall consistency)	. (root)
VBench++	Extends VBench with: • VBench‑I2V – image‑to‑video evaluation • VBench‑Long – metrics for long, Sora‑style videos • VBench‑Trustworthiness – fairness, bias, safety checks	vbench2_beta_i2v, vbench2_beta_long, vbench2_beta_trustworthiness
VBench‑2.0	Adds intrinsic faithfulness dimensions (commonsense reasoning, physics realism, human motion, creative composition, etc.) – 5 broad categories, 18 fine‑grained capabilities	VBench-2.0

How to get started

1. Install (Python 3.9+ recommended) – the package is on PyPI:

   pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118   # pick the CUDA version you need
   pip install vbench
   

2. Pick a benchmark – import the appropriate class (e.g., VBench, VBenchI2V, VBenchLong, VBenchTrustworthiness, or the VBench‑2.0 suite) and point it at a folder of generated videos.
3. Run evaluation – a single call computes all metrics for the selected dimensions and returns a dictionary of scores. The library also provides utilities to visualize radar charts and to compare against the public leaderboard.
4. Submit results – upload your scores (or generated videos) to the Hugging Face “VBench Arena” spaces linked in the README to see where your model ranks among the ~40‑plus models already listed.

Resources & community

• Paper: CVPR 2024 (VBench) and TPAMI 2025 (VBench++) – both linked in the README.
• Leaderboard & video arenas: Hugging Face spaces for VBench, VBench‑I2V, VBench‑2.0, etc.
• Dataset: A Google‑Drive folder with all sampled videos used for the benchmark (download link in the README).
• Citation: BibTex entries are provided for each version; cite the appropriate paper when using the benchmark.

Who should use it?

• Researchers developing new text‑to‑video, image‑to‑video, or long‑duration video generation models.
• Engineers building commercial video‑generation services who need objective quality reports.
• Anyone interested in tracking progress on video‑generation fairness, bias, and safety.

---

All information above is taken directly from the repository’s README; no external assumptions have been added.