MMMU-Benchmark/MMMU

What it solves

It addresses the lack of rigorous benchmarks for multimodal AI models that require expert-level, college-level knowledge and complex reasoning. Most existing benchmarks do not sufficiently challenge a model's ability to integrate domain-specific knowledge across diverse academic disciplines.

How it works

The project provides two distinct benchmarks:

• MMMU: A massive dataset of 11.5K multimodal questions across 30 subjects and 183 subfields (e.g., Art, Business, Science, Health, Humanities, and Engineering). It uses 32 different image types, including chemical structures and music sheets, to test advanced perception and reasoning.
• MMMU-Pro: A more stringent version that increases difficulty by filtering out questions answerable by text alone, adding plausible distractor options, and using vision-only inputs where questions are embedded within images to force the model to read and see simultaneously.

Who it’s for

Researchers and developers building next-generation multimodal foundation models who want to evaluate their models' performance against human experts and other state-of-the-art Large Multimodal Models (LMMs).

Highlights

• Expert-Level Scope: Covers college-level subject knowledge across six core disciplines.
• Diverse Visuals: Includes a wide variety of image types like charts, diagrams, and maps.
• Robust Evaluation: MMMU-Pro specifically targets intrinsic reasoning by removing text-only shortcuts.
• Comprehensive Splits: Provides development, validation, and test sets for systematic model tuning and evaluation.