tongjingqi/Thinking-with-Video

We introduce 'Thinking with Video', a new paradigm leveraging video generation for multimodal reasoning. Our VideoThinkBench shows that Sora-2 surpasses GPT5 by 10% on eyeballing puzzles and reaches 69% accuracy on MMMU.

What it solves

This project introduces "Thinking with Video," a reasoning paradigm that uses video generation models to solve complex multimodal problems. It aims to overcome the limitations of static image-based reasoning and the separation of text and vision by using dynamic video generation to visualize reasoning processes, such as drawing, imagination, and temporal evolution.

How it works

The project proposes using video generation models (like Sora-2) to perform reasoning by generating videos that represent the steps of a solution. To evaluate this, the authors created VideoThinkBench, a comprehensive benchmark consisting of:

  • Vision-Centric Tasks: Challenges involving spatial reasoning (eyeballing puzzles), pattern recognition (visual puzzles), abstract reasoning (ARC-AGI-2), and path-finding (mazes).
  • Text-Centric Tasks: Reasoning problems adapted from established benchmarks like GSM8K, MATH, and MMLU, where the model must embed the reasoning or answer within generated video frames.

Who it’s for

This is primarily for AI researchers and developers working on multimodal LLMs, video generation models, and the intersection of generative AI and cognitive reasoning.

Highlights

  • New Reasoning Paradigm: Shifts the focus from "Thinking with Text" (CoT) to "Thinking with Video."
  • VideoThinkBench: The first dedicated benchmark for evaluating the reasoning capabilities of video generation models.
  • Competitive Performance: Demonstrates that video generation models can surpass state-of-the-art vision-language models (VLMs) on specific vision-centric tasks like eyeballing puzzles.
  • Unified Modality: Shows that video models can handle both text-based reasoning and visual reasoning within a single generative framework.

Related

  • Project
    MME-Benchmarks/Video-MME-v2

    A robust benchmark for evaluating video understanding in MLLMs, using a three-level progressive difficulty scale and grouped non-linear scoring to measure true temporal reasoning.

  • Project
    Osilly/Vision-DeepResearch

    A framework and benchmark for developing Multimodal Large Language Models capable of deep research through iterative visual and textual search across images and videos.

  • Project
    deepgenteam/deepgen

    DeepGen 1.0 is a lightweight 5B parameter unified multimodal model that integrates image generation, editing, and reasoning into a single framework.

  • Project
    OpenGVLab/InternVideo

    A series of video foundation models and large-scale datasets designed for multimodal video understanding, long-context modeling, and contextual reasoning.

  • Project
    OpenSenseNova/SenseNova-Vision

    SenseNova-Vision is a unified multimodal model that treats diverse computer vision tasks as a generation problem, producing text, images, or mixed outputs for tasks like detection, segmentation, and depth estimation.