umbertogriffo/rag-chatbot

RAG (Retrieval-augmented generation) ChatBot that provides answers based on contextual information extracted from a collection of Markdown files.

What it solves

This project provides a local, conversation-aware chatbot that can answer questions based on a specific set of Markdown files. It solves the problem of generating accurate, context-aware responses from private documents while avoiding the need for external cloud APIs by running everything locally.

How it works

The system uses a Retrieval-Augmented Generation (RAG) pipeline combining llama.cpp for local LLM inference and Chroma as a vector database.

  1. Memory Building: Markdown files are split into chunks using a refactored RecursiveCharacterTextSplitter, converted into numerical embeddings via Sentence Transformers, and stored in Chroma.
  2. Incremental Indexing: To avoid rebuilding the entire index, the system tracks document versions using hashes and a SQLite mapping table, updating only changed or new documents.
  3. Retrieval: When a user asks a question, the LLM first rewrites the query for better retrieval. The system then fetches the most relevant chunks from the vector store.
  4. Synthesis: To prevent context overflow, the system employs two strategies: "Create and Refine" (sequential synthesis) or "Hierarchical Summarization" (combining independent answers).
  5. Conversation Memory: The chatbot saves chat history to maintain context across multiple interactions.

Who it’s for

It is designed for users who want to host their own private AI assistant that can "read" their documentation or personal notes (in Markdown format) and provide answers based on that specific knowledge base.

Highlights

  • Local Execution: Runs on CPU or GPU (CUDA/Metal) using quantized GGUF models via llama.cpp.
  • Incremental Updates: Efficiently updates the vector index without full rebuilds when documents change.
  • Flexible Synthesis: Supports multiple strategies to handle large amounts of retrieved context.
  • Query Rewriting: Improves retrieval accuracy by prompting the LLM to optimize the user's question before searching the database.
  • Full Stack: Includes a FastAPI backend and a React/TypeScript frontend with a document uploader.