opendatalab/OmniDocBench

[CVPR 2025] A Comprehensive Benchmark for Document Parsing and Evaluation

What it solves

OmniDocBench addresses the need for a comprehensive and high-quality benchmark to evaluate document parsing models in real-world scenarios. It provides a standardized way to measure how well AI models can handle diverse document types (like academic papers, financial reports, and handwritten notes) and complex elements (like nested tables and mathematical formulas) across different languages and layouts.

How it works

The project provides a dataset of 1,651 PDF pages with rich, multi-level annotations. It evaluates models across several dimensions:

  • Layout Detection: Identifying block-level elements (headings, tables, figures) and span-level elements (text lines, inline formulas).
  • Recognition: Assessing the accuracy of text OCR, formula recognition (using LaTeX), and table recognition (using LaTeX and HTML).
  • Reading Order: Verifying if the model correctly identifies the sequence of document components.
  • Metrics: It employs various metrics including Normalized Edit Distance, BLEU, METEOR, TEDS, and COCODet (mAP, mAR).

Who it’s for

It is designed for researchers and developers building document parsing systems, OCR engines, and multimodal LLMs that need to extract structured information from complex PDFs.

Highlights

  • Diverse Coverage: Includes 10 document types, 5 layout types, and 5 language types.
  • Deep Annotations: Features 28 block-level and 4 span-level annotations, including specific attribute tags for page and block characteristics.
  • High Quality: Data is curated through manual screening, intelligent annotation, and expert quality checks.
  • Flexible Evaluation: Supports both end-to-end and single-module evaluation with provided code and Docker-based runtime environments.

Related

  • Project
    run-llama/ParseBench

    ParseBench is an Apache‑2.0 benchmark that evaluates PDF‑parsing tools on five dimensions (tables, charts, content faithfulness, semantic formatting, visual grounding) using a 2 k‑page enterprise dataset. It ships a CLI, 180+ ready‑made pipelines (LlamaParse, OpenAI, Anthropic, Google, Azure, etc.), deterministic rule‑based metrics, and generates interactive HTML reports and leaderboards.

  • Project
    Yuliang-Liu/MultimodalOCR

    A collection of benchmarks (MDPBench, OCRBench, and OCRBench v2) used to evaluate the OCR and multilingual document parsing capabilities of Large Multimodal Models in real-world scenarios.

  • Dispatch
    Hugging Face Accelerating Document AI

    Hugging Face provides a comprehensive guide to using open-source multimodal models to automate document classification, parsing, and visual question answering for enterprise workflows.

  • Project
    PaddlePaddle/PaddleOCR

    PaddleOCR is an open‑source OCR and document‑AI toolkit that converts images or PDFs into structured, LLM‑ready text, tables and markdown/JSON. It offers multilingual scene‑text models (PP‑OCRv6) and lightweight vision‑language models (PaddleOCR‑VL, PP‑StructureV3) with state‑of‑the‑art accuracy, fast CPU/GPU inference, and deployment options ranging from Docker to browser SDKs. Integrated with RAG platforms like Dify and RAGFlow, it serves as a data‑engine for building AI agents and retrieval‑augmented generation pipelines.

  • Project
    docling-project/docling-parse

    Docling Parse is a fast, configurable PDF‑parsing library (Python API with a C++ core) that extracts text, words, lines, vector shapes and bitmap images together with their coordinates. It supports both sequential and multi‑threaded parsing, lets you choose what to compute or materialize per page, and includes rendering utilities. Ideal for AI workflows that need layout‑aware document data.