nashsu/llm_wiki

What it solves

LLM Wiki transforms static documents into a dynamic, interlinked personal knowledge base. Unlike traditional RAG systems that retrieve information on the fly, this project incrementally builds and maintains a persistent wiki, ensuring knowledge is compiled once and kept current rather than re-derived for every query.

How it works

The system uses a three-layer architecture: Raw Sources (immutable), a generated Wiki (LLM-created), and a Schema (rules and configuration). It employs a two-step Chain-of-Thought ingest process where an LLM first analyzes sources for entities and connections, then generates the actual wiki pages with source traceability.

Key technical components include:

• Knowledge Graph: A relevance model using direct links, source overlap, and community detection (Louvain algorithm) to visualize and cluster knowledge.
• Retrieval Pipeline: A multi-phase process combining tokenized search, optional vector semantic search (via LanceDB), and graph expansion to provide high-recall context to the LLM.
• Agentic Workflow: A Rust-based backend agent that can use tools for wiki/web search, workspace file generation, and executing approved shell commands.
• Deep Research: An automated pipeline that identifies knowledge gaps and uses web search (Tavily, SerpApi, SearXNG) to synthesize new research pages into the wiki.

Who it’s for

Researchers, students, and knowledge workers who want to move beyond simple chat-with-PDFs and build a structured, long-term digital garden or second brain that is automatically maintained by AI.

Highlights

• Two-Step Ingest: Improves quality by separating analysis from page generation.
• Knowledge Graph Insights: Automatically detects "surprising connections" and knowledge gaps in your data.
• Obsidian Compatibility: The generated wiki directory is compatible as an Obsidian vault.
• Multimodal Ingestion: Extracts images from PDFs and generates factual captions using vision LLMs.
• Deep Research: Automatically finds and ingests external web data to fill gaps in the existing knowledge base.
• Agent Skills: Supports local SKILL.md files to extend the agent's capabilities.