llm-wiki-compiler (llmwiki)

What it is – An open‑source CLI/SDK that turns raw documents (Markdown, PDFs, web pages, images, etc.) into a structured, citation‑aware markdown wiki. The wiki is compiled once by LLMs, then agents and humans can query, browse, lint, export, and reuse the knowledge without re‑reading the original files.

---

Core ideas

• Compile‑time knowledge – Implements Andrej Karpathy’s LLM Wiki pattern: LLMs extract concepts and generate typed pages (concept, entity, comparison, overview) once, storing provenance and citations.
• Configurable Lifecycle Profiles (CLP) – A JSON contract (.llmwiki/profile.json) defines entity schemas, relations, state‑machines, review gates, and retrieval policies. The compiler enforces these rules at runtime.
• Review & trust gates – Generated pages can be held for human review; policies enforce evidence, confidence, and schema checks before pages become live.
• Hybrid retrieval – Semantic chunk search + BM25 reranking + wikilink graph expansion builds compact evidence packs for queries.
• Open Knowledge Format (OKF) – Portable export/import of compiled wikis as markdown with structured front‑matter, plus JSON/JSON‑LD/GraphML/Marp.
• MCP server & SDK – Provides a Machine‑Control‑Protocol (MCP) API for agents and a TypeScript SDK (createWiki) for programmatic use.

---

Main commands (CLI)

Command	Purpose
llmwiki ingest <url-or-file>	Add a source file/URL to sources/.
llmwiki quickstart <source>	One‑step ingest + compile + open viewer.
llmwiki compile	Incrementally extract concepts and generate wiki pages.
llmwiki view	Launch a read‑only local web UI that refreshes after each compile.
llmwiki query "question"	Ask a grounded question against the compiled wiki; --save stores the answer as a page.
llmwiki lint	Validate citations, links, freshness, and policy compliance.
llmwiki eval	Produce health scores, citation coverage, and regression metrics.
llmwiki export --target <format> / import	Exchange knowledge via OKF, JSON, GraphML, etc.
llmwiki serve	Run an MCP server exposing ingest, compile, query, lint, export, etc.
`llmwiki template init <autosci	newsroom>`
llmwiki profile …	Create, show, validate, or diff the CLP contract.
llmwiki workflow …	List and run declared workflow steps (e.g., review, artifact creation).
llmwiki artifact write/verify	Manage hash‑pinned files declared by the profile.
llmwiki connector …	Run first‑party connectors (e.g., Crossref import).

---

When to use it

• You have a stable body of knowledge (papers, notes, transcripts) that you want to compile into a reusable, citation‑traceable knowledge base.
• You need agents to retrieve evidence packs quickly, without re‑processing raw files each time.
• You require auditability – every claim links back to source line ranges, and review queues enforce quality.
• You want to share the compiled wiki with other tools via OKF or standard graph/JSON formats.
• You are building a domain‑specific workflow (research, newsroom, etc.) and want a declarative schema that the system enforces.

Not a fit: a generic static‑site generator, a heavyweight ontology store, or a live‑search over rapidly changing logs.

---

Installation & quick start

# Global install (npm)
npm install -g llm-wiki-compiler

# Set up a provider (example: Anthropic)
export ANTHROPIC_API_KEY=sk-...
# or use OpenAI, Ollama, etc.

# One‑off compile and explore
llmwiki quickstart ./notes.md
llmwiki query "what are the key ideas?"
llmwiki view --open

The quickstart command creates a sources/ folder, runs the compile pipeline, and opens the local viewer.

---

Project layout (after init)

sources/          # raw files you ingest
wiki/              # compiled markdown pages (concepts/, queries/, etc.)
.llmwiki/          # internal state, profile.json, config, indexes, candidates
artifacts/         # hash‑pinned files declared by the profile
log.md             # activity journal

---

Extensibility

• Profiles let you declare new entity types, fields, and lifecycle states without writing code.
• Templates (autosci, newsroom) are signed, versioned bundles that scaffold a ready‑to‑use profile.
• SDK (createWiki({root})) gives programmatic access to all CLI operations from TypeScript.
• MCP makes the wiki callable from any language that implements the MCP protocol, enabling autonomous agents to ingest, query, and retrieve context packs.

---

License & community

• MIT licensed.
• CI badge shows automated tests run on each push.
• Documentation hosted at llmwiki.atomicstrata.ai.
• Issues and feature requests are handled via the GitHub repository.

---

In a nutshell: llm-wiki-compiler is a full‑stack toolchain for turning heterogeneous documents into a structured, citation‑rich markdown wiki that both humans and LLM‑based agents can reliably query and extend.