thiswillbeyourgithub/wdoc

wdoc – Retrieval‑Augmented Generation for Heterogeneous Documents

What it does – wdoc is a Python‑based RAG system that can ingest, index, and query a mix of document types (PDFs, EPUBs, Word, PowerPoint, audio/video, YouTube, Anki flashcards, web pages, etc.). It builds embeddings for the chunks, uses a two‑stage LLM pipeline (a cheap “evaluator” to filter retrieved chunks, then a strong LLM to generate a sourced answer) and finally merges the intermediate answers into a single markdown response. It also provides a high‑quality summarisation mode that produces detailed, indented markdown outlines of the source material.

Key components

• LangChain + LiteLLM – the core retrieval, chunking and LLM‑calling logic.
• Vector store – stores embeddings for fast similarity search; supports binary‑embedding stores for higher compression.
• Two‑stage LLM workflow – Eve (cheap model) filters retrieved chunks, Anna (strong model) extracts information, Carl merges results; optional Raphael re‑phrases queries and Sam summarises.
• Docker + Gradio UI – an optional web interface for users who prefer a GUI over the CLI.
• Extensive file‑type loaders – 15+ built‑in loaders (PDF, audio, video, YouTube, Anki, logseq markdown, etc.) plus recursive loaders (DuckDuckGo web search, JSON‑entries, link‑file).
• Provider‑agnostic LLM support – works with any model exposed through LiteLLM (OpenAI, Anthropic, Mistral, Ollama, OpenRouter, local Ollama servers, etc.). Private mode can run entirely offline.
• Caching & tracing – LLM/embedding calls are cached; optional Langfuse integration for observability.

Typical workflow

1. Load – wdoc --path <source> --filetype <type> downloads/parses the source and splits it into chunks.
2. Embed – each chunk is embedded using the selected embedding model.
3. Retrieve – a multi‑query retriever pulls the top‑k most similar chunks.
4. Filter – a cheap LLM judges relevance of each retrieved chunk.
5. Answer – a strong LLM extracts information from the relevant chunks, producing one answer per chunk.
6. Combine – answers are clustered semantically and merged hierarchically into a single, sourced markdown answer.
7. (Optional) Summarise – the same pipeline can be run in summarise mode to produce a detailed outline of the whole document.

How to use it

• CLI – uvx wdoc[full] --path <url_or_file> --task query --query "…" (or summarize). Extras ([full]) install all optional loaders.
• Docker UI – run the provided Docker compose; a Gradio front‑end lets you upload files, set the task, and view markdown results.
• Python library – import wdoc and call its high‑level API; the repository ships a wdoc‑skill folder with full reference tables and example snippets.

Why it matters

• Handles heterogeneous corpora in a single index, something most off‑the‑shelf RAG tools struggle with.
• The two‑stage LLM approach keeps costs low while preserving high recall.
• Provides sourced answers, making it easier to verify information.
• Extensible via plugins, custom loaders, and Open‑WebUI tool integration.

Current status – usable and tested, under active development with a roadmap for more features, better UI, and deeper private‑mode guarantees.