opendatalab/OmniDocBench

[CVPR 2025] A Comprehensive Benchmark for Document Parsing and Evaluation

OmniDocBench – A Benchmark for Document‑Parsing Models

What it is – OmniDocBench is a public dataset + evaluation suite for measuring how well AI systems can parse complex documents (PDFs). It targets the whole pipeline: detecting layout blocks, recognizing text, extracting tables, reading formulas, and preserving the correct reading order.

Why it matters – Modern multimodal models (e.g., OCR‑enabled vision‑language models) are often evaluated on narrow tasks like plain‑text OCR or table detection. Real‑world PDFs mix many element types, languages, and layouts. OmniDocBench provides a single, richly‑annotated benchmark that lets researchers compare end‑to‑end systems on all these aspects with a common set of metrics.

Key features (as described in the README)

  • Diverse content – 1 651 PDF pages covering 10 document categories (academic papers, financial reports, newspapers, textbooks, handwritten notes, etc.), 5 layout styles (single‑, double‑, triple‑column, mixed, other) and 5 language groups (English, Simplified Chinese, mixed, …).
  • Fine‑grained annotations –
    • Block‑level: 28 categories (title, paragraph, figure, table, equation, header/footer, code block, references, …).
    • Span‑level: 4 categories (text line, inline formula, footnote mark, ignored formula).
    • Each element stores its polygon coordinates, reading‑order index, and recognition ground truth (plain text, LaTeX for formulas, LaTeX + HTML for tables).
    • Additional attribute tags for pages, tables, text blocks and formulas (e.g., language, layout type, watermark, scan quality, table borders, formula type).
  • High annotation quality – Combination of manual screening, intelligent pre‑annotation, and expert + large‑model verification.
  • Evaluation code – Open‑source Python pipeline that can run end‑to‑end or module‑wise (layout detection, OCR, table recognition, formula recognition, reading‑order). It supports several established metrics:
    • Normalized Edit Distance, BLEU, METEOR for text.
    • TEDS and Edit Distance for tables.
    • CDM (Content‑Dependent Metric) and Edit Distance for formulas.
    • COCODet style mAP/mAR for layout detection.
  • Multi‑granularity adaptive matching (MGAM) – A matching algorithm that adapts the granularity of predictions to minimise bias, introduced in v1.6.
  • Easy deployment – Docker image with a reproducible environment (Python 3.10, TeX Live 2025, ImageMagick 7, Ghostscript) and a Conda‑based fallback.
  • Community integration – EvalScope support for running the benchmark against any OpenAI‑compatible endpoint.

Typical workflow

  1. Prepare ground‑truth JSON (the format shown in the README) and your model’s predictions in the same structure.
  2. Write a config YAML specifying which metrics to compute for each element type.
  3. Run the pipeline (python pdf_validation.py --config …) inside the Docker container or Conda env.
  4. Inspect results – overall scores plus per‑page and per‑attribute breakdowns, helping you pinpoint where your model struggles (e.g., handwritten notes, dense formulas, rotated tables).

Who would use it

  • Researchers building multimodal vision‑language models, OCR engines, or specialized document‑understanding systems.
  • Companies that need to benchmark their document‑automation pipelines (invoice processing, academic literature mining, financial report analysis).
  • Anyone looking for a standardised, multilingual, multi‑layout test set to report results in papers or product documentation.

Where to get it

  • Dataset: Hugging Face hub (opendatalab/OmniDocBench) and OpenDataLab.
  • Code & docs: This GitHub repository (opendatalab/OmniDocBench).
  • Paper: arXiv link provided in the header.

TL;DR – OmniDocBench is a genuine, actively maintained benchmark for comprehensive document‑parsing evaluation, offering a large, multi‑type PDF dataset with detailed annotations and ready‑to‑run evaluation scripts.

Related

  • Project
    run-llama/ParseBench

    A benchmark for evaluating how well document parsing tools convert PDFs into structured output that AI agents can reliably use for autonomous decision-making.

  • Project
    Yuliang-Liu/MultimodalOCR

    A collection of benchmarks (MDPBench, OCRBench v2, and OCRBench) designed to evaluate the OCR and document parsing capabilities of Large Multimodal Models across multiple languages and real-world scenarios.

  • Dispatch
    Hugging Face Accelerating Document AI

    Hugging Face provides a comprehensive guide to using open-source multimodal models to automate document classification, parsing, and visual question answering for enterprise workflows.

  • Project
    PaddlePaddle/PaddleOCR

    PaddleOCR is an open‑source OCR and document‑AI toolkit that converts images/PDFs into structured, LLM‑ready JSON or Markdown. It offers multilingual scene‑text models (PP‑OCRv6), a lightweight vision‑language parser (PaddleOCR‑VL‑1.6) for tables, formulas, and ancient scripts, and a structure‑aware pipeline (PP‑StructureV3). Models run on CPU, GPU, XPU, NPU or via ONNX/TensorRT, and integrate with RAG/agent platforms like Dify and RAGFlow.

  • Project
    Vchitect/VBench

    VBench is an open‑source benchmark suite that automatically evaluates video‑generation models across dozens of quality dimensions (technical, aesthetic, and trustworthiness). It includes prompt collections, metric pipelines, human‑aligned scores, and public leaderboards for T2V, I2V, long‑video, and intrinsic‑faithfulness evaluation.