Yuliang-Liu/MultimodalOCR

On the Hidden Mystery of OCR in Large Multimodal Models (OCRBench)

What it solves

This project addresses the lack of comprehensive, real-world evaluation tools for the Optical Character Recognition (OCR) and document parsing capabilities of Large Multimodal Models (LMMs). It specifically targets gaps in evaluating multilingual support, low-resource languages, non-Latin scripts, and the ability to handle photographed (rather than just digital) documents.

How it works

The repository provides a suite of three specialized benchmarks:

  • MDPBench: Focuses on multilingual document parsing across 17 languages and diverse scripts, testing models on both clean digital pages and real-world photographed documents.
  • OCRBench v2: A large-scale bilingual benchmark covering 31 diverse scenarios (such as receipts, formulas, and diagrams) with 10,000 human-verified QA pairs to test visual text localization and reasoning.
  • OCRBench: An initial comprehensive benchmark assessing five core areas: text recognition, scene-text VQA, document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.

Who it’s for

It is designed for researchers and developers building or evaluating Large Multimodal Models to ensure their systems can accurately parse text and documents across various languages and real-world visual conditions.

Highlights

  • Multilingual Scope: MDPBench covers 17 languages, including Arabic, Hindi, Thai, and Vietnamese.
  • Real-World Testing: Specifically evaluates performance drops when moving from digital documents to photographed ones.
  • Diverse Scenarios: OCRBench v2 includes 31 different scenarios, including street scenes and technical diagrams.
  • Human-Verified: Datasets utilize rigorous pipelines of expert labeling and manual correction for high-quality ground truth.

Related

  • Project
    opendatalab/OmniDocBench

    A comprehensive benchmark for evaluating document parsing in real-world scenarios, featuring rich annotations for layout detection, OCR, table, and formula recognition.

  • Project
    Yuliang-Liu/MonkeyOCRv2

    A visual-text foundation model for Document AI that provides high-accuracy multilingual document parsing, understanding, and a standalone vision encoder.

  • Project
    MMMU-Benchmark/MMMU

    A massive multi-discipline benchmark for evaluating multimodal AI models on college-level subject knowledge and complex reasoning across 30 different subjects.

  • Project
    MME-Benchmarks/Video-MME-v2

    A robust benchmark for evaluating video understanding in MLLMs, using a three-level progressive difficulty scale and grouped non-linear scoring to measure true temporal reasoning.

  • Dispatch
    Visual Salamandra 7B Release

    Hugging Face's Language Technologies Lab has released Visual Salamandra, a 7-billion parameter multimodal model that extends the Salamandra LLM to support images and video with a focus on European linguistic diversity.