CinePile 2.0 release: adversarial refinement boosts video QA dataset quality

TL;DR

CinePile 2.0 is a major upgrade of the long‑video question‑answering dataset that uses an adversarial refinement pipeline to turn weak or degenerate QA pairs into high‑quality, vision‑dependent questions, and the release includes the full pipeline code and a refreshed benchmark that shows substantial gains for both commercial and open‑source video‑LLMs.

What is CinePile?

CinePile is a long‑video QA dataset built from YouTube movie clips and audio descriptions for the visually impaired. The original May 2024 release contained ~300 k training samples and 5 k test samples. It was distinguished by:

  • Question diversity – covering temporal reasoning, plot analysis, character dynamics, setting details, and thematic exploration.
  • Question difficulty – human annotators outperformed the best commercial vision models by 25 % and open‑source models by 65 % on the benchmark.

Data creation pipeline (CinePile 1.0)

  1. Template mining – questions from MovieQA and TVQA were clustered with the UAE‑Large‑V1 similarity model. Ten random examples per cluster were fed to GPT‑4 to generate a template and a prototypical question.
  2. Template selection – Gemini 1.0 Pro chose the most relevant templates for each clip.
  3. Scene‑specific generation – a language model received the scene transcript, selected template names, example questions, and a system prompt. The prompt forced the model to use timestamps and to provide rationales, which improved answer plausibility and distractor quality.
  4. Scale – about 32 questions were generated per video.
  5. Quality filters – three LLMs (Gemini, GPT‑3.5, Phi‑1.5) flagged degenerate questions (answer obvious from the question alone). Gemini also scored whether a question required visual information. Difficulty was measured by testing models with full context.

Limitations of the first release

  • Degeneracy filtering discarded many questions that still contained useful video information.
  • The filter only covered the test split because running multiple proprietary models on the full training set was too costly.
  • Some questions could be answered without any visual or dialogue context, reducing the benchmark’s focus on visual reasoning.

Adversarial Refinement: Turning Weak QA into Strong QA

The new adversarial refinement pipeline replaces outright discarding with iterative improvement:

  1. Deaf‑Blind LLM – a model that sees only the question and answer choices (no transcript or visual data). CinePile 2.0 uses LLaMA 3.1 70B for this role.
  2. Rationale extraction – the Deaf‑Blind LLM returns an answer and a textual rationale, exposing implicit cues.
  3. Question modification – GPT‑4 rewrites the question and/or answer options to eliminate the identified cues.
  4. Iterative loop – steps 1‑3 repeat up to five times until the Deaf‑Blind LLM’s accuracy drops to random chance (≈20 %).
  5. Robustness checks – all five permutations of answer order are tested; a question is marked degenerate if the model succeeds in three or more permutations.

Impact on the dataset

Metric Test set Training set
Degenerate pairs fixed 90.24 % 90.94 %
Unfixable pairs (manual review) ~80 of 800 retained (no negative impact)

The pipeline therefore salvages the majority of weak items, preserving valuable video content while ensuring that the final benchmark truly requires visual understanding.

Implementation details

  • Code release – the full adversarial refinement pipeline, including prompts, is open‑source at https://github.com/JARVVVIS/Adversarial-Refinement.
  • Local execution – LLaMA 3.1 70B runs locally, avoiding API rate limits and cloud costs.
  • Prompt design – prompts guide GPT‑4 to focus on removing implicit clues without altering the underlying semantics of the question.

Evaluations on CinePile 2.0

The revised test set was evaluated with previously benchmarked models and 16 new video‑LLMs. Key findings:

  • Gemini 1.5 Pro leads commercial VLMs, especially in Setting and Technical Analysis (environmental and character‑interaction questions).
  • GPT‑based models excel in Narrative and Plot Analysis.
  • Gemini 1.5 Flash achieved 58.75 % overall accuracy, with strong performance on setting‑related categories.

Open‑source model progress

Model Accuracy Notable observation
LLaVa‑One Vision 49.34 % Massive jump from Video‑LLaVA’s 22.51 % on CinePile 1.0
LLaVa‑OV (7B) – Competitive with larger 26 B models
MiniCPM‑V 2.6 (8B) – Outperformed InternVL2 (26 B)

Even the best open‑source models lose 15‑20 % accuracy on the hard split, confirming that CinePile 2.0 remains a challenging benchmark for visual narrative comprehension.

Hard‑split analysis

The hard split isolates questions that demand deeper visual reasoning. All evaluated models show a pronounced accuracy drop, underscoring a substantial gap between current AI systems and human performance on complex video understanding tasks.

Leaderboard and community involvement

A live CinePile 2.0 leaderboard (https://huggingface.co/spaces/tomg-group-umd/CinePileLeaderboard) accepts new model submissions and updates continuously, providing a transparent platform for tracking progress.

Why CinePile 2.0 matters

  • Dataset quality at scale – adversarial refinement demonstrates a practical method for upgrading existing datasets without manual curation.
  • Benchmark relevance – the refined test set forces models to rely on visual information, making performance gains more meaningful.
  • Open‑source accessibility – publishing the pipeline and code enables other researchers to apply the same technique to their own multimodal datasets.
  • Community signal – the leaderboard and improved open‑source results show rapid progress, yet the hard‑split gap highlights ample room for future research.

The authors of the post are Ruchit Rawal, Miquel Farré, Gowthami Somepalli, and Leandro von Werra from Hugging Face.

Sources