HUST-AI-HYZ/MemoryAgentBench

What it solves

It provides a standardized benchmark to evaluate how well LLM agents handle memory across incremental, multi-turn interactions. It specifically addresses the need to measure an agent's ability to remember, update, and resolve conflicts in information provided over a long conversation, rather than just processing a single long prompt.

How it works

MemoryAgentBench simulates real-world AI assistant conversations by splitting data into chunks to mimic multi-turn interactions. It uses an "inject once, query multiple times" design to improve efficiency. The benchmark evaluates agents across four core competencies:

• Accurate Retrieval (AR): Finding specific information from memory.
• Test-Time Learning (TTL): Learning new information during the interaction.
• Long-Range Understanding (LRU): Understanding context over long distances.
• Conflict Resolution (CR): Handling contradictory information.

Who it’s for

Researchers and developers building LLM agents, RAG systems, or agentic memory methods who need to quantify their model's memory performance and reliability in multi-turn scenarios.

Highlights

• Multi-Turn Simulation: Data is chunked to simulate realistic daily conversations.
• Diverse Datasets: Includes newly constructed datasets like EventQA and FactConsolidation, alongside reformulated data from other benchmarks.
• Efficiency: One long text can be used for multiple questions to reduce evaluation overhead.
• Comprehensive Metrics: Supports various metrics including exact match, substring match, Recall@5, and LLM-as-judge evaluation.