thiswillbeyourgithub/wdoc

Summarize and query from a lot of heterogeneous documents. Any LLM provider, any filetype, advanced RAG, advanced summaries, scriptable, etc

What it solves

wdoc is a RAG (Retrieval-Augmented Generation) system designed to handle large, heterogeneous collections of documents. It solves the problem of querying and summarizing vast amounts of information across diverse file types (such as PDFs, audio, video, and Anki flashcards) while ensuring that answers are sourced and grounded in the actual content to prevent hallucinations.

How it works

The system uses a multi-stage pipeline to ensure high recall and specificity:

  1. Retrieval: Documents are retrieved using embeddings and multi-query retrievers.
  2. Evaluation: A "weak" LLM (Eve the Evaluator) filters out irrelevant documents from the initial retrieval.
  3. Answering: A "strong" LLM (Anna the Answerer) extracts relevant information from each remaining document.
  4. Aggregation: A final LLM (Carl the Combiner) aggregates these individual answers into a single markdown-formatted response. To improve coherence, answers are grouped using semantic clustering (via scipy's hierarchical clustering) before being combined.

For summaries, the system processes text in chunks, using a strong LLM to create detailed, logically indented summaries while maintaining context from previous chunks.

Who it’s for

It is designed for researchers, students, and professionals who need to extract definitive, sourced answers from extensive and diverse information sources.

Highlights

  • Broad File Support: Supports 15+ file types, including PDFs, EPUBs, Word, PowerPoint, YouTube videos, audio files, and Anki collections.
  • Extensible Architecture: Works as both a CLI tool and a Python library, with a Gradio-based Docker web UI and an Open-WebUI tool integration.
  • Privacy-Focused: Includes a "private mode" that censors outgoing connections and avoids API keys to ensure data remains local.
  • Advanced PDF Parsing: Employs 15 different loaders and a scoring system to select the best parser for any given PDF, including table support.
  • Web Search Integration: Preliminary support for querying the web via DuckDuckGo.
  • Sourced Answers: Every answer is linked back to the document hash for verification.