supermemoryai/memorybench

What it solves

MemoryBench provides a standardized way to evaluate and compare memory and context systems (such as long-term memory providers for AI). It solves the difficulty of testing different memory providers side-by-side using consistent datasets and judging criteria, preventing the need to write custom evaluation code for every new tool.

How it works

The framework uses a pluggable pipeline that separates the memory provider, the benchmark dataset, and the judge model. The process follows these steps:

1. Ingest: Loads benchmark sessions into the provider.
2. Index: Waits for the provider to index the data.
3. Search: Queries the provider to retrieve relevant context.
4. Answer: Generates an answer using a specified LLM.
5. Evaluate: A judge model compares the answer to the ground truth to assign a score.
6. Report: Aggregates results into a final report.

It features checkpointing, allowing users to resume failed runs from the last successful stage, and a Web UI for real-time inspection of failures and questions.

Who it’s for

It is designed for developers and researchers building or integrating AI memory systems who need to quantitatively measure accuracy, latency, and token usage across different providers.

Highlights

• Interoperable Architecture: Mix and match any supported provider, benchmark dataset, and judge model (e.g., GPT-4o, Claude, Gemini).
• Multi-provider Comparison: Run the same benchmark across multiple providers simultaneously to compare performance.
• MemScore: A composite metric that tracks quality (accuracy), latency (ms), and context tokens to highlight performance tradeoffs.
• Pluggable Design: Users can easily add their own custom benchmarks, providers, or judges.