Ayanami0730/deep_research_bench

What it solves

DeepResearch Bench addresses the lack of a systematic way to evaluate Deep Research Agents (DRAs). It provides a standardized framework to measure how well these agents can handle complex, PhD-level research tasks across various professional domains.

How it works

The benchmark uses 100 expert-crafted research tasks across 22 fields (such as science, finance, and software). It employs two primary evaluation methodologies:

1. RACE (Reference-based Adaptive Criteria-driven Evaluation): Assesses report quality by dynamically generating task-specific criteria (comprehensiveness, insight, instruction-following, and readability) and comparing the agent's output against high-quality reference reports.
2. FACT (Framework for Factual Abundance and Citation Trustworthiness): Evaluates grounding and retrieval by extracting factual claims and their citations, then using web scraping and LLM judgment to verify if the cited sources actually support the claims.

Who it’s for

AI researchers and developers building deep research agents who need a rigorous, expert-validated benchmark to measure their agent's analytical depth, factual accuracy, and report-writing quality.

Highlights

• Expert-Curated: Tasks designed by PhD-level experts and senior practitioners based on real-world user query distributions.
• Multi-Dimensional Scoring: Evaluates both the qualitative aspects of the final report and the quantitative accuracy of its citations.
• Broad Domain Coverage: Spans 22 distinct fields, including STEM, business, and humanities.
• Automated Pipeline: Includes scripts for cleaning articles and running evaluations using frontier LLMs as judges.