RUC-NLPIR/FlashRAG

⚡ FlashRAG – A Python Toolkit for Efficient Retrieval‑Augmented Generation (RAG) Research

What it is – FlashRAG is an open‑source Python library that bundles together everything needed to reproduce, benchmark, and extend state‑of‑the‑art RAG systems. It ships with:

36 pre‑processed RAG benchmark datasets.
23 ready‑made RAG algorithms (including 7 that combine retrieval with reasoning).
A modular framework for plugging in retrievers, rerankers, generators, and compressors.
Optional acceleration back‑ends such as vLLM, FastChat, and Faiss.
A lightweight web UI (FlashRAG‑UI) for configuring pipelines and visualising results.

Core Components

Component	Role	Notable Options
Retriever	Finds relevant documents. Supports dense (sentence‑transformers + Faiss), sparse (BM25 via Pyserini or `bm25s`), and neural sparse (SPLADE via Seismic). Also includes a Web‑search retriever (Serper API).
Reranker	Re‑orders the retrieved list; can be swapped with any model that scores query‑doc pairs.
Generator	The LLM that produces the final answer. Supports OpenAI models, multimodal LLMs (LLaVA, Qwen, InternVL) and any HuggingFace model via vLLM.
Compressor	Reduces long documents (e.g., chunking with the external Chunkie library).
Pipeline	Orchestrates the above pieces. Two built‑in pipelines are provided – a simple `SequentialPipeline` and a base class for custom pipelines.

Highlighted Features

Extensive benchmark suite – 36 datasets are already pre‑processed and hosted on HuggingFace/ModelScope, so you can start evaluating immediately.
23 SOTA RAG algorithms – one‑click configuration of methods such as RQRAG, Adaptive‑RAG, Spring, Trace, IRCoT, and the newer reasoning pipelines (R1‑Searcher, Search‑R1, etc.).
Reasoning‑based pipelines – combine chain‑of‑thought or graph‑based reasoning with retrieval, achieving ~60 F1 on multi‑hop tasks like HotpotQA.
Multimodal support – retrieve with CLIP‑based visual retrievers and generate with vision‑language models.
Speed‑focused tooling – optional vllm for fast LLM inference, Faiss for dense vector indexes, and a lightweight bm25s fallback for BM25.
UI – a browser‑based interface lets you drag‑and‑drop components, run experiments, and view evaluation metrics without writing code.

Quick‑Start (Python)

# Install the development package (includes optional deps)
pip install flashrag-dev[full]   # or `pip install flashrag-dev --pre`

from flashrag.config import Config
from flashrag.pipeline import SequentialPipeline
from flashrag.utils import get_dataset

# Load a config (YAML file or dict) that points to a dataset and selects components
cfg = Config(config_file_path='my_config.yaml')

dataset = get_dataset(cfg)['test']
pipe = SequentialPipeline(cfg)
results = pipe.run(dataset, do_eval=True)

The results object contains per‑question answers, retrieved passages, and evaluation scores (e.g., EM, F1).

When Would You Use FlashRAG?

Research – reproduce published RAG papers, compare new ideas against a common benchmark suite, or benchmark a novel retriever/generator combo.
Prototype – assemble a custom RAG pipeline (e.g., multimodal retrieval + LLM) with only a few lines of code.
Education – the UI and extensive docs make it easy for students to explore how each component affects performance.
Benchmarking – the bundled 36 datasets cover QA, fact‑checking, and multi‑hop reasoning, providing a one‑stop shop for systematic evaluation.

Installation Notes

Python ≥ 3.10 is required.
Faiss must be installed via conda (CPU‑only or GPU version) because pip wheels are not universally compatible.
Optional extras:
- vllm for high‑throughput LLM inference.
- sentence‑transformers for dense retrieval.
- pyserini if you prefer the original BM25 implementation.
For SPLADE (neural sparse) you need a Rust toolchain and the pyseismic‑lsr package.

Documentation & Resources

Paper: FlashRAG: A Python Toolkit for Efficient RAG Research (arXiv:2405.13576) – accepted to ACM WWW 2025 Resource Track.
Datasets: Hosted on HuggingFace and ModelScope (36 pre‑processed splits).
UI Demo: https://github.com/user-attachments/assets/8ca00873-5df2-48a7-b853-89e7b18bc6e9
GitHub: https://github.com/RUC-NLPIR/FlashRAG (MIT license)

TL;DR

FlashRAG is a ready‑to‑use, extensible toolkit for anyone working on Retrieval‑Augmented Generation. It bundles datasets, dozens of SOTA methods, fast indexing/retrieval back‑ends, and a visual UI, making it straightforward to reproduce papers, benchmark new ideas, or build production‑grade RAG pipelines.