yixuantt/MultiHop-RAG
Repository for "MultiHop-RAG: A Dataset for Evaluating Retrieval-Augmented Generation Across Documents" (COLM 2024)
What it solves
It addresses the difficulty of evaluating RAG pipelines when answers are not contained in a single document. Many real-world scenarios require "multi-hop" reasoning, where a system must retrieve and connect information from multiple different documents and their metadata to answer a query correctly.
How it works
MultiHop-RAG provides a specialized QA dataset containing 2,556 queries. For every query, the necessary evidence is spread across 2 to 4 different documents. The project includes scripts to test retrieval performance (with and without reranking) and QA accuracy, allowing developers to benchmark how well their system handles complex, multi-document information retrieval.
Who it’s for
It is designed for AI researchers and developers building RAG pipelines who need a rigorous way to evaluate their system's ability to perform multi-hop retrieval and reasoning.
Highlights
- Contains 2,556 complex queries requiring evidence from 2-4 documents.
- Includes document metadata to simulate real-world RAG complexities.
- Provides ready-to-use scripts for retrieval testing, QA testing, and evaluation.
- Published as a peer-reviewed paper accepted by COLM 2024.