DocAILab/XRAG

📚 What is XRAG?

XRAG (eXamining the Core – Benchmarking Foundational Component Modules in Advanced Retrieval‑Augmented Generation) is an open‑source Python framework that lets researchers and developers evaluate the building blocks of Retrieval‑Augmented Generation (RAG) systems. Instead of treating a RAG pipeline as a black box, XRAG breaks it into modular pieces—retrievers, embeddings, text splitters, orchestrators, etc.—and runs systematic benchmarks so you can see how each choice affects overall performance.

---

✨ Core Features (as described in the README)

Category	Highlights
Evaluation Engine	• Multiple evaluation dimensions: traditional metrics (F1, EM, MRR, Hit@K, MAP, NDCG) and LLM‑based metrics (faithfulness, relevance, correctness). • Deep evaluation metrics such as contextual precision/recall, hallucination, bias. • Built‑in evaluators: LlamaIndex, DeepEval, plus hooks for custom metrics.
Modular Architecture	• Pluggable components for retrievers, embeddings, LLMs. • Supports vector, BM25, hybrid, tree‑structured, keyword, and document‑summary retrieval. • Easy to add custom retrieval or evaluation strategies.
LLM Integration	• OpenAI API, local models via Ollama, Hugging Face (e.g., Qwen, LLaMA). • Configurable model name, temperature, etc., via config.toml.
Orchestrators / Agentic RAG	Five orchestrator types (sequential, conditional, iterative, parallel, hybrid) that manage the flow of retrieval, post‑processing, and generation, enabling experiments like self‑RAG, adaptive‑RAG, FLARE, etc.
User Interfaces	• Command‑line tool xrag-cli with sub‑commands (run, webui, generate, api). • Interactive Web UI for dataset upload, index building, pipeline configuration, live query testing, and visual report generation.
Utility Commands	• generate – automatically creates QA pairs from a document folder. • api – launches a lightweight HTTP service exposing /query and /health endpoints.
Extensible Config	All settings live in a simple config.toml (API keys, model choices, chunk size, dataset name, etc.) with an --override flag for quick experiments.

---

🛠️ Installation (quick recap)

1. Create a Python 3.11+ environment (conda recommended).

   conda create -n xrag python=3.11
   conda activate xrag
   

2. Install the package from PyPI (the package name is examinationrag).

   pip install examinationrag
   # install jury without pulling extra deps (required by XRAG)
   pip install jury --no-deps
   # pin a few transitive deps to avoid conflicts
   pip install requests==2.27.1 urllib3==1.25.11 jiwer<4.0.0
   

3. Verify the CLI is available:

   xrag-cli --help
   

---

🚀 Quick‑Start Walk‑through

1. Edit config.toml – set your API keys, choose a model (openai, ollama, or huggingface), pick an embedding model, and select a dataset (e.g., hotpot_qa).
2. Run a benchmark:

   xrag-cli run
   

   You can override any config entry on the fly, e.g.:

   xrag-cli run --override embeddings="BAAI/bge-base-en-v1.5" llm="ollama"
   

3. Launch the interactive Web UI (optional but handy for visual inspection):

   xrag-cli webui
   

   The UI walks you through dataset upload, index construction, pipeline definition, live querying, and finally shows a detailed evaluation report.
4. Expose XRAG as a service (useful for integration with other apps):

   xrag-cli api --host 0.0.0.0 --port 8000 --dataset_folder ./my_docs
   

   Then POST to http://localhost:8000/query with a JSON payload containing query and top_k.
5. Generate synthetic QA data from a folder of documents:

   xrag-cli generate -i ./docs -o ./qa.json -n 200 -s 30
   

---

🎯 Who Might Use XRAG?

Audience	Typical Use‑Case
Academic researchers	Systematically compare how different retrievers, embeddings, or orchestrators affect RAG performance on standard benchmarks (HotpotQA, DropQA, NaturalQA).
Product engineers	Validate that a new retrieval component (e.g., a tree‑based index) meets latency and accuracy targets before shipping.
LLM‑centric startups	Generate internal QA datasets automatically and expose a /query API for rapid prototyping of RAG‑powered assistants.
Students / hobbyists	Learn about RAG pipelines by toggling components in the Web UI and observing metric changes in real time.

---

📖 Documentation & Community

• Paper – Accepted at ICDE 2026 (arXiv 2412.15529). The README links to the PDF for deeper methodological details.
• GitHub – Issues, pull‑requests, and a star counter are visible; the repo encourages contributions (code, data, ideas).
• Acknowledgements – Built on LlamaIndex, Hugging Face Transformers, and inspired by projects such as RAGLAB, FlashRAG, FastRAG, AutoRAG, and LocalRAG.
• Citation – A ready‑to‑copy BibTeX entry is provided for academic credit.

---

🛎️ TL;DR

XRAG is a modular benchmarking suite for Retrieval‑Augmented Generation. It lets you mix‑and‑match retrievers, embeddings, LLM back‑ends, and orchestrators, then evaluates the result with both classic IR metrics and newer LLM‑based quality scores. Install via pip install examinationrag, configure with a simple TOML file, and run experiments either from the command line or an interactive Web UI.