Mnemo: Local-First AI Memory Layer for LLMs

Mnemo is a local-first AI memory layer that enables Large Language Models (LLMs) to maintain persistent memory across sessions by extracting entities and relationships into a structured knowledge graph. By operating as a sidecar service, Mnemo allows developers to inject scored and ranked context into LLM prompts without relying on cloud-based memory services or Python runtimes.

Core Architecture and Workflow

Mnemo functions as a sidecar service that processes conversation data to build a persistent knowledge graph. The system is built using Rust, leveraging SQLite for storage and the petgraph crate for graph operations.

The Ingestion Pipeline

When raw text (such as a conversation turn or a document) is sent to the /ingest endpoint, Mnemo performs the following steps:

  1. Entity Extraction: The configured LLM identifies named entities (people, tools, places, concepts) and the relationships between them.
  2. Deduplication: Entities are deduplicated based on name and type, and aliases are merged.
  3. Persistence: Data is written to SQLite, and the in-memory petgraph is updated atomically.

The Retrieval Pipeline

When a request is made to the /retrieve endpoint, Mnemo executes a six-stage pipeline to generate a context_prompt:

  1. Full-text chunk search: Initial search for relevant text blocks.
  2. Entity name search: Identification of specific mentioned entities.
  3. Graph expansion: Breadth-First Search (BFS) over the knowledge graph to find related entities.
  4. Relation filter: Filtering based on established relationships.
  5. Scoring and Ranking: Results are ranked; graph-expanded results are weighted at 0.5x to ensure direct matches rank higher than inferred ones.
  6. Assembly: The final ranked context is assembled into a string for injection into the LLM's system prompt.

Technical Specifications and Performance

Mnemo is distributed as a single static Rust binary, eliminating the need for a Python runtime or vendor lock-in. It supports any OpenAI-compatible backend, including fully local options like Ollama, as well as OpenAI and Anthropic.

Performance Benchmarks

Benchmarked on an Apple M2 with SQLite in WAL mode, the system demonstrates high throughput and low latency (debug build numbers; release builds are 3–5–x faster):

Operation Avg Latency Throughput
Entity insert (SQLite) ~0.12 ms ~8,300 ops/s
Entity lookup by ID ~0.08 ms ~12,500 ops/s
Chunk insert ~0.14 ms ~7,100 ops/s
Full-text chunk search ~0.28 ms ~3,500 ops/s
Graph neighbor (depth=1) ~0.21 ms ~4,700 ops/s
Full retrieval pipeline ~4.2 ms ~238 ops/s

Comparison with Alternative Memory Layers

Mnemo distinguishes itself from typical AI memory tools through its runtime and storage approach. While many alternatives rely on Python daemons or cloud-based storage, Mnemo uses a single Rust binary and local SQLite storage.

Unlike naive context dumps, Mnemo uses a graph layer (petgraph) for multi-hop traversal, allowing it to retrieve related information that might not be explicitly mentioned in the current query but is logically connected via the knowledge graph.

Deployment and Integration

Mnemo provides multiple integration paths for developers:

  • Docker: A recommended path using Docker Compose to deploy Mnemo alongside Ollama for a fully free, local setup.
  • Binary: Direct installation via Cargo for use with existing Ollama or OpenAI instances.
  • Python SDK: A mnemo-sdk package for easy integration into Python-based LLM pipelines.

Community Insights and Considerations

Discussion among developers highlights several critical considerations for implementing local memory layers:

  • Context Pollution: Some users note that filling a model's context window with too many memories can potentially degrade model performance.
  • Fact Conflict: Questions have been raised regarding how the system handles contradictory facts (e.g., an entity changing jobs), which is a key challenge for any persistent knowledge graph.
  • Integration: There is a consensus that while standalone memory layers are useful for custom pipelines, many managed agent harnesses are beginning to integrate these features natively.

Sources