ConTextual: Benchmarking Multimodal Reasoning in Text-Rich Scenes

Researchers from the University of California Los Angeles (UCLA) have introduced ConTextual, a specialized dataset and leaderboard designed to evaluate the ability of Large Multimodal Models (LMMs) to perform context-sensitive text-rich visual reasoning. This benchmark addresses a critical gap in current LMM evaluations, which often focus on simple instruction following rather than the joint reasoning required to understand text within a visual context, such as navigating maps or interpreting memes.

Dataset Composition and Scope

ConTextual consists of 506 challenging instructions paired with text-rich images. The dataset is designed to ensure that tasks require joint reasoning over both textual and visual cues to reach a correct answer. It covers eight real-world visual scenarios:

  • Time Reading
  • Shopping
  • Navigation
  • Abstract Scenes
  • Mobile Applications
  • Webpages
  • Infographics
  • Miscellaneous Natural Scenes

The dataset is provided in two formats: a validation set of 100 instances (including instructions, images, and reference answers) and a test dataset containing only instructions and images.

Experimental Results and Model Performance

Initial experiments assessed 13 models across three categories: Augmented LLMs (GPT-4 combined with OCR/captions), Closed-Source LMMs (GPT-4V and Gemini-Vision-Pro), and Open-Source LMMs (including LLaVA-v1.5-13B, ShareGPT4V-7B, and Idefics-9B).

Evaluation Methodology

Evaluation is conducted using an "LLM-as-a-judge" approach. GPT-4 is used as the judge to determine if a predicted response is acceptable compared to a human-written reference response, as this method showed the highest correlation with human judgment.

Key Performance Findings

  • General Struggle: Both proprietary and open-source LMMs struggle with the ConTextual dataset compared to human performance, indicating a significant opportunity for model improvement.
  • Domain Gaps: Proprietary models, including GPT-4V, perform poorly in infographic reasoning and time-reading tasks. While GPT-4V outperforms humans in abstract reasoning—likely due to exposure to memes and quotes—it fails in time-related tasks where humans excel.
  • Open-Source Limitations: Models like LLaVA-1.5-13B and ShareGPT-4V-7B show a strong performance gap between abstract/natural scenes and other domains like shopping, navigation, and mobile usage, suggesting these domains are out-of-distribution for these models.
  • Failure of Augmented LLMs: Augmenting a standard LLM with OCR or image captions results in poor performance, achieving a human approval rate of only 17.2%. This highlights that these tasks require precise visual perception and fine-grained vision-language alignment rather than simple text conversion.

Future Directions for LMM Development

To improve context-sensitive text-rich visual reasoning, the researchers suggest focusing on three primary technical areas:

  1. Enhanced Image Encoders: Developing more powerful encoders to better capture visual detail.
  2. Accurate Image Descriptions: Creating higher-fidelity descriptions of visual elements.
  3. Fine-grained Vision-Language Alignment: Improving the alignment between visual and textual tokens to mitigate hallucinations and improve perception.

Submission Guidelines for the Community

Researchers can submit their models to the ConTextual leaderboard via the following methods:

  • Validation Set: Use the auto-evaluation pipeline on GitHub and submit results in a JSON format mapping image URLs to boolean scores (0 or 1) via the Hugging Face leaderboard form.
  • Test Set: Send model predictions (mapping image URLs to predicted responses) directly to the research team via email for evaluation.

Sources