atomicstrata/llm-wiki-compiler

The knowledge compiler. Raw sources in, interlinked wiki out. Inspired by Karpathy's LLM Wiki pattern.

What it solves

llmwiki transforms raw, unstructured data (PDFs, notes, web pages, transcripts) into a structured, interlinked Markdown wiki. Unlike traditional RAG (Retrieval-Augmented Generation), which retrieves raw chunks at query time, llmwiki compiles knowledge once into durable, citation-traceable pages that agents and humans can browse and query. This prevents the need to re-discover relationships between documents every time a question is asked.

How it works

The system uses a two-phase LLM pipeline to extract concepts and generate typed pages (such as concepts, entities, and comparisons). It implements a "compile-time" approach to knowledge management:

  1. Ingestion: Raw sources are collected in a sources/ directory.
  2. Compilation: The LLM processes these sources into interlinked Markdown files with YAML frontmatter, storing them in a wiki/ directory.
  3. Lifecycle Profiles: Users can define a .llmwiki/profile.json to declare typed entities, relations, and workflows, ensuring the generated knowledge follows a specific domain schema (e.g., research or editorial).
  4. Retrieval: It uses a hybrid approach combining semantic chunk search, BM25 reranking, and wikilink graph expansion to provide context for queries.

Who it’s for

  • AI Agents: Developers building agents that need a stable, citation-aware context pack rather than a pile of loose files.
  • Researchers and Editors: People managing large volumes of domain-specific material who need an auditable, structured knowledge base.
  • Knowledge Engineers: Those looking to implement the "LLM Wiki" pattern for persistent, compounding knowledge.

Highlights

  • Citation Traceability: Every claim and paragraph cites specific source files and line ranges, validated by a built-in linter.
  • Configurable Lifecycle Profiles (CLP): Allows the creation of domain-specific knowledge substrates with enforced entity schemas and trust gates.
  • MCP Server: Exposes the wiki's functionality (ingest, compile, query, eval) to MCP-compatible agents.
  • Open Knowledge Format (OKF): Supports importing and exporting knowledge as portable Markdown bundles.
  • Quality Gates: Includes an evaluation harness (llmwiki eval) and review policies to hold generated pages for human approval.

Related

  • Project
    nvk/llm-wiki

    llm‑wiki is a cross‑runtime tool that lets Claude, Codex, OpenCode, Pi and other LLM agents build and query a structured, Obsidian‑compatible wiki. It provides ingestion, parallel research, session memory, private adapters, and exportable knowledge checkpoints, with install scripts for each supported agent.

  • Project
    nashsu/llm_wiki

    A desktop application that automatically transforms documents into a structured, interlinked personal wiki using LLMs, featuring a knowledge graph and automated deep research.

  • Project
    SamurAIGPT/llm-wiki-agent

    A coding agent skill that converts source documents into a persistent, interlinked markdown wiki with automated entity extraction and contradiction flagging.

  • Project
    xoai/sage-wiki

    A graph-based knowledge base that uses LLMs to compile raw documents into an interlinked wiki and knowledge graph for shared human-AI memory.

  • Project
    ussumant/llm-wiki-compiler

    A Claude Code and Codex plugin that compiles scattered markdown files and codebases into a synthesized, topic-based wiki to reduce AI agent context costs.