GraphRAG-Bench/GraphRAG-Benchmark

The official repo of GraphRAG-Bench for evaluating GraphRAG models. "When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation". (ICLR'26)

What it solves

GraphRAG-Bench addresses the lack of clarity regarding when Graph Retrieval-Augmented Generation (GraphRAG) actually outperforms traditional RAG. While GraphRAG is designed to handle hierarchical structures and complex reasoning better, it often underperforms vanilla RAG in real-world tasks. This benchmark provides a systematic way to identify the specific scenarios and conditions where graph structures provide measurable benefits.

How it works

The project provides a comprehensive evaluation framework that tests models across a full pipeline—from graph construction and knowledge retrieval to final generation. It utilizes a dataset with tasks of increasing difficulty across two domains (literary/fictional and medical/healthcare), measuring performance across four dimensions:

  1. Fact Retrieval: Basic accuracy and ROUGE-L.
  2. Complex Reasoning: Ability to connect disparate pieces of information.
  3. Contextual Summarization: Accuracy and coverage of summaries.
  4. Creative Generation: Factual scores and coverage for generative tasks.

Who it’s for

This tool is for AI researchers and developers building RAG systems who need to determine whether a graph-based approach is more effective than a traditional vector-based approach for their specific data and use case.

Highlights

  • Multi-level Difficulty: Tasks range from simple fact retrieval to complex creative generation.
  • Domain-Specific Leaderboards: Includes dedicated benchmarks for both medical and literary content.
  • Full-Pipeline Evaluation: Standardized evaluation code to ensure fair comparison across different GraphRAG frameworks.
  • Framework Support: Provides instructions and examples for integrating and evaluating various GraphRAG frameworks.