run-llama/ParseBench

ParseBench – A Benchmark for Document‑Parsing Tools used by AI Agents

What it is

• An open‑source benchmark (Apache‑2.0) that measures how well PDF‑parsing systems turn raw pages into structured data that downstream AI agents can rely on.
• Focuses on five capability dimensions that matter for production workflows: Tables, Charts, Content Faithfulness, Semantic Formatting, and Visual Grounding.
• Comes with a curated, human‑verified dataset of ~2 000 enterprise pages (insurance, finance, government) hosted on HuggingFace.

Why it matters

• Agents often need to read tables, extract chart data, or follow visual cues (e.g., strikethrough prices). A parser that looks correct but loses structure or hallucinates content can break downstream decisions.
• The benchmark provides deterministic, rule‑based metrics (no LLM‑as‑judge by default) so scores are reproducible and comparable across commercial APIs, open‑weight models, and custom pipelines.

Key components

Component	What it does
Dataset	llamaindex/ParseBench on HuggingFace; split into five JSONL files, each with ground‑truth format and a set of evaluation rules.
Pipelines	180+ pre‑defined configurations that wrap specific parsing services (LlamaParse, OpenAI, Anthropic, Google Gemini, Azure Document Intelligence, AWS Textract, etc.). New pipelines can be added via a simple Claude‑code integration command.
CLI	parse-bench (installed via pip install "parse-bench[runners]" or from source). Commands include run, download, status, pipelines, compare, leaderboard, and serve.
Metrics	• Tables – GTRM (GriTS + TableRecordMatch) • Charts – ChartDataPointMatch • Content Faithfulness – Content Faithfulness Score • Semantic Formatting – Semantic Formatting Score • Visual Grounding – Element Pass Rate.
Reports	Generates per‑dimension HTML dashboards, CSV/JSON/Markdown summaries, and an aggregated leaderboard.

Typical workflow

1. Install – pip install "parse-bench[runners]" (or uv sync --extra runners).
2. Configure – Put the needed API keys in a .env file (e.g., LLAMA_CLOUD_API_KEY, OPENAI_API_KEY).
3. Download data – uv run parse-bench download (or --test for a tiny 3‑file‑per‑category subset).
4. Run a pipeline – uv run parse-bench run llamaparse_agentic (add --test for a quick sanity check).
5. View results – uv run parse-bench serve llamaparse_agentic opens an interactive browser report.
6. Compare – uv run parse-bench compare <pipeline_a> <pipeline_b> produces side‑by‑side HTML.

Extensibility

• New providers/pipelines can be registered with a single Claude‑code command (/integrate-pipeline <name> <API docs>), which scaffolds the provider class and updates docs.
• Optional fast extra enables a JIT‑accelerated TEDS table metric (identical scores, faster for large tables).

Leaderboard (excerpt)

Rank	Provider	Overall	Tables	Charts	Content Faith.	Sem. Formatting	Visual Ground.	Cost per page
1	LlamaParse Agentic Plus	90.20	93.37	94.18	92.25	87.12	84.09	5.62¢
2	LlamaParse Agentic	87.01	88.88	88.68	91.78	81.44	84.25	1.25¢
…	…	…	…	…	…	…	…	…
(Full, sortable leaderboard is at https://parsebench.ai.)

Citation

@misc{zhang2026parsebench,
  title={ParseBench: A Document Parsing Benchmark for AI Agents},
  author={Boyang Zhang and Sebastián G. Acosta and Preston Carlson and Sacha Bron and Pierre‑Loïc Doulcet and Daniel B. Ospina and Simon Suo},
  year={2026},
  eprint={2604.08538},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2604.08538}
}

Links

• Website: https://parsebench.ai
• Paper: https://arxiv.org/abs/2604.08538
• Dataset: https://huggingface.co/datasets/llamaindex/ParseBench
• Code: https://github.com/run-llama/ParseBench
• Companion benchmark (schema‑guided extraction): https://github.com/run-llama/ExtractBench