run-llama/ParseBench

What it solves

ParseBench addresses the difficulty of evaluating document parsing tools. Most existing evaluations focus on how similar the output looks to a reference text, but ParseBench tests whether the parsed output preserves the structural and semantic meaning necessary for AI agents to make reliable autonomous decisions based on PDFs.

How it works

It uses a dataset of approximately 2,000 human-verified pages from real enterprise documents (finance, insurance, and government). The benchmark evaluates parsing tools across five specific capability dimensions:

• Tables: Checks structural fidelity of merged cells and hierarchical headers.
• Charts: Extracts exact data points, series, and axis labels.
• Content Faithfulness: Detects omissions, hallucinations, and reading-order errors.
• Semantic Formatting: Ensures meaning-carrying formatting (like strikethroughs or bold text) is preserved.
• Visual Grounding: Verifies that extracted elements can be traced back to their original location on the page.

Evaluation is primarily deterministic and rule-based rather than relying on LLM-as-a-judge.

Who it’s for

Developers building AI agents that rely on PDF parsing for production workflows and researchers developing new document parsing models or APIs.

Highlights

• Agent-Centric Evaluation: Focuses on functional utility for AI agents rather than simple text similarity.
• Comprehensive Dataset: Includes 1,211 documents across five distinct failure modes.
• Extensive Pipeline Support: Comes with 90+ pre-configured pipelines for evaluating various commercial and open-weight parsing tools.
• Detailed Reporting: Generates interactive HTML reports, aggregation dashboards, and side-by-side comparisons.