Manifold — Gaussian-Clef RAG

What it is

A Python library (3.10+) for retrieval-augmented generation (RAG) — the technique where an AI searches a document collection for relevant passages before answering a question. The twist: instead of representing each text chunk as a single point in a high-dimensional space (the standard approach), it represents each chunk as a Gaussian distribution — a "cloud" with a center (average meaning) and a spread (uncertainty). Retrieval becomes a question of how much two clouds overlap.

It runs 100% locally with zero API keys required for basic use. Optional cloud integrations exist for extra features.

The core idea

Every text chunk is mapped to a probability distribution 𝒩(μ, Σ):

• μ (mean) — semantic meaning, from a local SentenceTransformer or Gemini embeddings
• Σ (covariance) — directional uncertainty, modeled as a diagonal plus low-rank decomposition (rank 8 by default)

Search scores how much probability mass the query's distribution places on each document's distribution — a metric called the Expected Likelihood Kernel (ELK).

Why it's fast

Comparing two high-dimensional distributions normally requires inverting large matrices (O(d³) operations). This project uses the Woodbury matrix identity and Sylvester determinant theorem to reduce that to O(d·r²) where r=8 — turning a 150ms operation into under 3ms. Combined with other optimizations, the project claims sub-100ms end-to-end retrieval.

Key components

Component	What it does	Claimed impact
Woodbury ELK Acceleration	Math trick for fast kernel evaluation	150ms → <3ms
Cloudflare Clef Decision Routing	Uses Cloudflare's AI model to detect ambiguous queries and ask clarifying questions	1,500ms → <40ms
Dual-Store Ingestion	Indexes every chunk twice: dense vector store + SQLite full-text search (FTS5)	Sub-2ms candidate search
Taxonomy Branch Pruning	Hierarchical clustering skips irrelevant branches before reranking	Prunes 60–80% of candidates
Clef 4-Question Passage Gate	Parallel filter for relevance, evidence, and contradictions	Prunes ~30 candidates → top 5–8
Universal Ingestion Engine	Accepts raw text, Markdown, JSON/JSONL, CSV, files, and directories	Zero preprocessing boilerplate
Flexible Corpus Generator	Cleans messy text (HTML, OCR scans, chat transcripts) into structured corpora with caching	Instant clean corpus creation

How to use it

from gaussian_rag import GaussianRAG

rag = GaussianRAG.create()
rag.add_texts(["Passage 1", "Passage 2", "Passage 3"])

results = rag.search("your question", top_k=3)
answer = rag.query("your question")  # optional LLM-generated answer

Also supports:

• Ingestion: add_markdown(), add_records() (JSON/JSONL), add_file(), add_directory(), add_raw_text() (dirty text auto-cleaned)
• Persistence: rag.save("path") and GaussianRAG.load("path")
• CLI: python -m gaussian_rag.main ingest/query, plus a corpus generator script (data/generate_corpus.py)
• Evaluation: python test_48_laws_e2e.py validates the full pipeline on The 48 Laws of Power; pytest runs test suites including latency benchmarks

Optional integrations

• Cloudflare Workers AI — enables Clef decision routing and passage gating (set CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN)
• OpenAI / Gemini / Mistral — for LLM answer generation (set respective API keys)

Configuration lives in config/config.yaml (embedding backend, dimension, low-rank size, retrieval metric, chunking parameters).

Extras

• 3D dashboard: FastAPI backend (port 8001) + Vite frontend (port 5173) showing interactive Gaussian ellipsoids, live stage-by-stage latency telemetry via Server-Sent Events, and Clef decision traces
• Documentation: extensive docs in docs/ covering architecture, design decisions, theory, and usage

License

GNU Affero General Public License v3.0 (AGPL-3.0).