NirDiamant/Controllable-RAG-Agent

What it solves

This project addresses the limitations of traditional RAG systems that rely on simple semantic similarity, which often fail to answer complex, multi-hop questions. It provides a controllable autonomous agent capable of performing multi-step reasoning and adaptive planning to answer non-trivial questions from custom datasets while strictly preventing hallucinations by grounding answers solely in the provided data.

How it works

The agent uses a deterministic graph as its "brain" to orchestrate a multi-stage process:

1. Data Processing: PDFs are loaded, split into chapters, cleaned, and summarized. Both the raw content, chapter summaries, and specific quotes are encoded into vector stores.
2. Question Anonymization: To avoid LLM bias and pre-trained knowledge, the agent replaces named entities in the question with variables to create a general execution plan.
3. Task Decomposition: The anonymized plan is de-anonymized and broken down into specific retrievable or answerable tasks.
4. Execution and Distillation: For each task, the agent decides whether to retrieve information from vector stores (distilling the results for accuracy) or generate a response using chain-of-thought reasoning.
5. Verification and Re-planning: The agent verifies that generated content is grounded in the original context. If verification fails, it updates its plan and repeats the process.
6. Final Answer: A final response is generated using the accumulated context and chain-of-thought reasoning.

Who it’s for

Developers and AI engineers building production-grade RAG systems that require high reliability, complex reasoning over large documents (like books), and strict adherence to source data to avoid hallucinations.

Highlights

• Deterministic Graph Reasoning: Uses a structured graph to manage complex autonomous agent behavior.
• Adaptive Planning: Continuously updates the execution plan based on new information retrieved during the process.
• Hallucination Prevention: Implements verification loops and grounding checks to ensure answers are based only on provided data.
• Entity Anonymization: Prevents LLM bias by planning based on variables rather than specific named entities.
• Ragas Evaluation: Integrated quality assessment using metrics like faithfulness, answer correctness, and context recall.