TimeScope: A New Benchmark for Long-Video Large Multimodal Model Understanding

Hugging Face has released TimeScope, an open-source benchmark designed to measure the actual temporal understanding of vision-language models (LMMs) when processing long videos. By inserting short "needle" clips into base videos ranging from 1 minute to 8 hours, TimeScope evaluates whether models truly comprehend sequences of events or are merely performing surface-level visual retrieval.

The Need for Temporal Benchmarking

Many current vision-language models claim to support massive context windows capable of handling thousands of frames. However, Hugging Face notes that these claims are often overstated because training data is frequently capped at 256 frames per clip, leading to significant performance degradation as video length increases.

Existing benchmarks, such as Video Needle in a Haystack (VideoNIAH), often use static images as needles, which measures visual search rather than temporal dynamics. TimeScope addresses this gap by using short video clips (5-10 seconds) as needles, forcing models to process temporal information and motion rather than just identifying a static frame.

TimeScope Benchmark Design

TimeScope utilizes a "needle-in-a-haystack" approach where one or more hand-curated short video needles are spliced into a long base video (e.g., a lecture or documentary). The benchmark evaluates three specific capabilities:

1. Localized Retrieval

This task tests the model's ability to spot and answer questions about a specific short segment within a long video. Success in this category typically requires the model to sample a relevant frame from the needle to identify a specific object or event.

2. Information Synthesis

This task requires the model to identify multiple text-based needles (such as "secret words" displayed on-screen) dispersed throughout the video and report them in the correct chronological order. This evaluates the model's ability to scan the entire timeline and understand relative positioning.

3. Fine-Grained Temporal Perception

This task focuses on motion and sequences that cannot be solved with single-frame sampling. Models must perceive dynamics across multiple frames—for example, counting how many times a person swings an axe—to provide the correct answer. This probes whether long-context handling preserves temporal fidelity.

Evaluation Results and Key Findings

Initial evaluations of leading vision-language models, including Gemini 2.5-Pro and various versions of Qwen 2.5-VL and InternVL 2.5, revealed several critical insights:

  • Temporal Horizon Limits: Most models experience "performance cliffs" as video duration increases. Gemini 2.5-Pro was the only model tested that maintained strong accuracy on videos longer than one hour.
  • Parameter Scaling vs. Temporal Ability: Increasing model size does not automatically extend the temporal horizon. For example, Qwen 2.5-VL 3B and 7B, as well as InternVL 2.5 models (2B, 4B, and 8B), showed nearly identical long-video performance curves, plateauing at roughly the same context length.
  • Task-Specific Trade-offs: Models exhibit different strengths across the three pillars. Qwen 2.5-VL performed well in Information Synthesis (OCR tasks) but struggled with Fine-Grained Temporal Perception (motion counting).

Open Source Availability

TimeScope is fully open-sourced to encourage community benchmarking and model improvement. The following resources are available:

  • Dataset: Hosted at Apollo-LMMs/TimeScope on Hugging Face.
  • Leaderboard: Available via the Apollo-LMMs/TimeScope Hugging Face Space.
  • Evaluation Framework: Integrated into the lmms-eval GitHub repository.

Sources