DocAILab/XRAG

XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation

📚 What is XRAG?

XRAG (eXamining the Core – Benchmarking Foundational Component Modules in Advanced Retrieval‑Augmented Generation) is an open‑source benchmarking framework for Retrieval‑Augmented Generation (RAG) systems. It lets researchers and developers measure how individual RAG components – retrievers, embeddings, LLMs, chunkers, orchestrators, etc. – affect overall performance.

✨ Core Features (as described in the README)

Feature	What it does
Comprehensive evaluation	Runs multiple evaluation dimensions (LLM‑based, deep‑evaluation, classic IR metrics) and reports metrics such as F1, EM, MRR, Hit@K, MAP, NDCG, plus LLM‑based faithfulness, relevance, correctness, hallucination, bias, etc.
Modular architecture	Plug‑and‑play components for retrievers, embeddings, LLMs; supports vector, BM25, hybrid, tree‑based, keyword, and summary‑based retrieval.
Multiple LLM back‑ends	Works with OpenAI APIs, local models via Ollama, Hugging‑Face models (e.g., Qwen, LLaMA).
Orchestrators / Agentic RAG	Five orchestrator types – sequential, conditional, iterative, parallel, hybrid – to build more complex, agent‑like RAG pipelines (self‑RAG, adaptive‑RAG, FLARE, etc.).
Web UI	`xrag-cli webui` launches an interactive dashboard for dataset upload, index building, pipeline configuration, live query testing, and visual evaluation reports.
CLI & API	Command‑line tool (`xrag-cli`) for running benchmarks, generating QA pairs, and starting a REST API server (`xrag-cli api`).
Configuration via TOML	All settings (API keys, model choices, chunk size, dataset, etc.) live in a `config.toml` file; can be overridden on the fly with `--override`.
Extensible evaluation models	Built‑in evaluators from LlamaIndex, DeepEval, plus custom metrics; roadmap includes novel metrics like Semantic Perplexity, Entropy, Auto‑J, Prometheus.

🚀 Getting Started (quick‑start flow)

Install – Python 3.11+, create a conda env, then pip install examinationrag (plus a few pinned dependencies).
Configure – edit config.toml (API keys, model, dataset, chunk size, etc.).
Run – xrag-cli run executes the full benchmark; add --override key=value to tweak parameters without editing the file.
Explore – launch the UI with xrag-cli webui to upload datasets, build indexes, define a RAG pipeline, and see live results.
Serve – start the backend service via xrag-cli api and query it with a simple POST /query HTTP call.

🛠️ Typical Use Cases

Scenario	How XRAG helps
Research paper – comparing retrieval strategies	Swap retriever modules (BM25 vs. vector) and instantly see impact on metrics like MRR and hallucination.
LLM evaluation – measuring faithfulness of generated answers	Use built‑in LLM‑based metrics or plug in a custom evaluator to score answer correctness.
Prototype an agentic RAG pipeline	Build a workflow with the “adaptive‑RAG” orchestrator, test it in the Web UI, and export the configuration.
Dataset creation	`xrag-cli generate` creates synthetic QA pairs from a document folder for downstream training.
Service deployment	Run `xrag-cli api` to expose a REST endpoint that other apps can call for retrieval‑augmented answering.

📦 Installation & Runtime Requirements

Python ≥ 3.11
Optional: conda environment recommended
Dependencies include jury, requests==2.27.1, urllib3==1.25.11, jiwer<4.0.0 (pinned to avoid conflicts)
Supports both cloud LLM APIs (OpenAI) and local models via Ollama or Hugging Face.

📚 Where to Learn More

Paper – XRAG: Examining the Core (ICDE 2026) – arXiv 2412.15529, DOI 10.1109/ICDE65706.2026.00201
GitHub – https://github.com/DocAILab/XRAG (stars, issues, roadmap)
Web UI Demo – run locally with xrag-cli webui
Citation – ready‑to‑copy BibTeX in the README.

🙋‍♀️ Who Might Use It?

Academic researchers studying retrieval, prompting, or evaluation of LLM‑augmented systems.
ML engineers building production RAG pipelines who need systematic component benchmarking.
Students looking for a hands‑on framework to experiment with different retrievers, embeddings, and orchestrators.

All information above is taken directly from the repository’s README; no external assumptions have been added.