MME-Benchmarks/Video-MME-v2

Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

What it solves

Video-MME-v2 is a benchmark designed to evaluate the true video understanding capabilities of multimodal large language models (MLLMs). It addresses the gap between high leaderboard scores on existing benchmarks and the actual user experience by introducing a more robust and progressive evaluation system that prevents models from achieving high scores through "scattered hits" or lucky guesses.

How it works

The benchmark decomposes video understanding into three progressive levels of difficulty:

  1. Level 1 (Multi-Point Information Aggregation): Tests the ability to retrieve and integrate multimodal cues (frames, audio, subtitles) from across a video.
  2. Level 2 (Temporal Understanding): Evaluates the ability to capture state changes, action sequences, and causal relations.
  3. Level 3 (Temporal Complex Reasoning): Tests the ability to combine temporal information with external world knowledge and social commonsense.

To ensure robustness, it uses a grouped non-linear scoring mechanism. Questions are organized into groups of four interrelated questions. Scores are based on either "Relevance" (consistency of answers within a group) or "Logic" (coherence of the reasoning chain), rewarding models that can consistently and logically answer related questions correctly.

Who it’s for

This project is for researchers and developers of video-based multimodal LLMs who need a rigorous way to measure their models' temporal reasoning and multimodal fusion capabilities.

Highlights

  • Three-Level Progression: Systematically tests from simple information retrieval to complex temporal reasoning.
  • Grouped Non-Linear Scoring: Reduces sensitivity to random correctness and better reflects model stability and robustness.
  • Linguistic Context Support: Provides options for concatenated or interleaved subtitles to test how text modality aids reasoning.
  • Comprehensive Dataset: Includes 800 videos and 3,200 human-annotated question-answer pairs, created with over 3,300 human-hours of expert annotation.
  • Flexible Evaluation Pipeline: Supports integration with VLMEvalKit, lmms-eval, and EvalScope, as well as a standalone Transformers-based script.

Related

  • Project
    tongjingqi/Thinking-with-Video

    A new multimodal reasoning paradigm and benchmark (VideoThinkBench) that evaluates the ability of video generation models to solve complex visual and textual reasoning tasks.

  • Project
    MCG-NJU/VideoChat3

    VideoChat3 is a 4B generalist video MLLM designed for efficient video understanding, covering everything from subtle motion and long-form reasoning to live stream proactive responses.

  • Project
    cambrian-mllm/cambrian-s

    A suite of multimodal LLMs and datasets designed to enhance spatial reasoning and "spatial supersensing" in video understanding.

  • Project
    OpenGVLab/InternVideo

    A series of video foundation models and large-scale datasets designed for multimodal video understanding, long-context modeling, and contextual reasoning.

  • Project
    Osilly/Vision-DeepResearch

    A framework and benchmark for developing Multimodal Large Language Models capable of deep research through iterative visual and textual search across images and videos.