elstongun/leviathan

Leviathan – Fast, token‑efficient memory for AI agents

Leviathan is a single‑binary Rust tool that turns any tabular or JSON‑like data source (CSV/TSV, JSONL, SQLite, or the output of a database CLI) into a full‑text searchable SQLite index. An agent can then ask natural‑language questions and receive a short, citation‑rich “card” containing the most relevant records – typically a few hundred tokens instead of millions.

---

What it does

Feature	What you get
Universal ingest	leviathan index reads JSONL, CSV, TSV, SQLite files, or any command that streams JSON rows (e.g., psql … -c "SELECT row_to_json(t) FROM …").
Schema mapping	A tiny leviathan.toml tells Leviathan which field is the record id, which fields are title, text, group, date, etc. Only id is mandatory.
Full‑text + BM25 ranking	Records are stored in an SQLite FTS5 table; queries are ranked by BM25 with optional per‑field boosts.
Group‑scoped search	-g <group> limits results to a customer, project, or any logical bucket; unknown groups are reported instead of guessed.
Token‑budgeted answers	The tool returns a fixed‑size “card” (≈ 450 tokens) that fits comfortably into LLM context windows.
Incremental updates	upsert and delete modify the index atomically, keeping it fresh without a full rebuild.
Agent integration	Comes with ready‑made skills for Claude, Cursor, VS Code/Copilot, Gemini CLI, etc.; a wrapper (leviathan wrap <agent>) generates the configuration needed for those agents.
Optional memory layer	A second SQLite file can store key‑value “memories” that agents can remember and recall within a token budget.
MCP / HTTP server	leviathan mcp exposes the same tools over a stdio‑based Multi‑Client‑Protocol; leviathan serve --http provides a REST/OpenAPI endpoint with bearer‑token or OAuth2.1 auth.
Zero runtime deps	The index is a plain SQLite file; the binary is self‑contained.

---

Typical workflow

1. Create a schema – leviathan init ./export generates a commented leviathan.toml you can edit.
2. Build the index – leviathan index tickets.csv --id "Ticket ID" --group customer_id --date created_at (or pipe JSON from any DB).
3. Query – leviathan search -g acme "sso login loop after password reset" returns a concise, cited card.
4. Hook into an agent – copy skills/leviathan into the agent’s skill directory or run leviathan wrap claude --memory --apply to add a tool that the LLM can call.

---

How it works (high‑level)

1. Ingestion – Rows are streamed, each field is tokenised, and the record is inserted into a SQLite file that contains an FTS5 virtual table.
2. Query processing – The tool resolves the requested group (exact match → name → fuzzy), builds a single FTS5 query that includes group and filter tokens, and runs it.
3. Ranking – Results are scored by BM25 × any user‑specified boosts (e.g., title weighted 2×).
4. Card generation – Only the top N records are decoded, truncated to a token budget, and formatted with citations, dates, and any display fields.
5. Memory (optional) – A separate SQLite file stores key‑value pairs; each write replaces the previous value for the same key, keeping the memory consistent.

---

Benchmarks (from the repo)

• 1 M records (≈ 678 MB) → median answer size 436 tokens vs. a naïve grep that would need > 100 k tokens.
• Relevant record recall – 99 % of the top‑5 results contain the correct record; rank‑1 recall is 98.5 %.
• Latency – median 33 ms per query (vs. 92 ms for the best grep strategy).
• Memory token budget – after 300 sessions a 33 K‑token markdown memory file is reduced to ~ 736 tokens when loaded.

---

Who might use it?

• LLM‑powered support agents that need fast, token‑efficient access to large ticket or log databases.
• RAG pipelines where the retrieval step must stay within a tight context window.
• Developers of autonomous agents (Claude, Gemini, ChatGPT plugins, etc.) who want a plug‑and‑play “knowledge base” without running a separate vector store.
• Data‑heavy teams that already store logs in CSV/SQLite and want a zero‑dependency, offline searchable index.

---

Getting started

# Install the binary from crates.io (or directly from Git)
cargo install leviathan-index
# Or download a pre‑built release from the GitHub Releases page.

# Index a CSV file
leviathan index tickets.csv --id "Ticket ID" --group customer_id --date created_at

# Search
leviathan search -g acme "sso login loop after password reset"

---

Limitations & considerations

• Full‑text only – No semantic embeddings; relevance is based on keyword matching and BM25.
• Schema‑driven – You must provide a mapping (id, title, etc.); fields not mapped are ignored.
• Single‑node – The index lives in one SQLite file; it is not a distributed store.
• No model inference – Leviathan does not run any LLM; it merely supplies concise context for a model to consume.

---

License & contribution

Leviathan is released under the Apache‑2.0 license. Contributions are welcomed via pull requests; benchmark‑affecting changes must include before/after numbers. Security concerns are addressed in SECURITY.md (offline by default, read‑only indexes unless explicitly served).

---

Bottom line: Leviathan gives you a fast, token‑lean way to turn any tabular dataset into a searchable memory that LLM agents can call directly, without the overhead of a vector database or external services.