atomicstrata/llm-wiki-compiler

The knowledge compiler. Raw sources in, interlinked wiki out. Inspired by Karpathy's LLM Wiki pattern.

What it solves

llmwiki transforms raw, unstructured data (PDFs, web pages, notes, transcripts) into a structured, interlinked markdown wiki. Unlike traditional RAG (Retrieval-Augmented Generation), which retrieves raw chunks at query time, llmwiki compiles knowledge once into durable, typed pages with citations. This prevents agents and humans from having to re-discover relationships in raw files repeatedly, creating a stable and auditable knowledge base that compounds over time.

How it works

The system uses a two-phase LLM pipeline to extract concepts and generate typed pages (such as concepts, entities, and comparisons). It implements a "compile-time" approach to knowledge: sources are ingested and processed into a wiki structure stored as markdown files with YAML frontmatter.

Key architectural components include:

  • Configurable Lifecycle Profiles (CLP): A JSON-based contract that defines entity schemas, relations, and trust gates, ensuring the runtime enforces domain-specific rules rather than relying on prompt conventions.
  • Hybrid Retrieval: Combines semantic chunk search, BM25 reranking, and wikilink graph expansion to build context packs for queries.
  • Quality Gates: Includes a linting system to validate citations and links, and an evaluation harness to measure health scores and citation precision.
  • Integration Layers: Provides a CLI, a TypeScript SDK, and an MCP (Model Context Protocol) server to allow agents to interact with the wiki.

Who it’s for

It is designed for users who need a persistent, auditable knowledge base from raw materials, such as researchers (using the autosci template) or editorial teams (using the newsroom template), as well as developers building AI agents that require stable, citation-aware context instead of loose files.

Highlights

  • Citation-Traceable Output: Every claim and paragraph cites specific source files and line ranges.
  • Domain Templates: Pre-built profiles for research and editorial workflows that define specific entity types and lifecycles.
  • MCP Server: Allows AI agents to ingest, compile, query, and manage the wiki directly.
  • Open Knowledge Format (OKF): Supports importing and exporting knowledge as portable markdown bundles.
  • Incremental Compilation: Only processes changed sources to save time and API costs.
  • Review Policy: Allows generated pages to be held for human approval based on confidence or provenance rules.