AutoTrustAI/PaperGuru-Benchmark

What it solves

PaperGuru addresses the limitation of current LLM agents in long-horizon tasks—such as writing 200K-token surveys or reproducing complex research papers—where standard RAG or context windows are insufficient. It introduces a "Lifecycle-Aware Memory" (LAM) primitive to prevent agents from using stale information, handle multi-hop relevance across citations, and maintain constant query costs as archives grow.

How it works

The system uses a Capital Chunk Memory (CCM) architecture that splits memory into two layers:

1. Chunk heads: A compact routing surface (one per artifact) used for efficient searching.
2. Chunk contents: The full raw text, accessed only when needed.

It organizes information via a temporal artifact graph with structural edges (e.g., cites, implements) and historical-causality edges (e.g., superseded-by, retracted-by). The processing pipeline follows a Search $\rightarrow$ Extract $\rightarrow$ Reason $\rightarrow$ Verify flow, producing provenance-grounded "evidence cards" that the agent uses to compose, critique, and mutate its output.

Who it’s for

Researchers and developers building long-horizon LLM agents for academic research, software engineering, or clinical evidence synthesis who need a memory system that tracks versioning and provenance.

Highlights

• State-of-the-Art Performance: Outperforms baselines on PaperBench (paper-to-code reproduction) and SurveyBench (long-form survey writing).
• Lifecycle Awareness: Specifically designed to handle revisions, deprecations, and retractions of information.
• Provenance Grounding: Every claim in the output is traceable back to a verifiable artifact in memory.
• Causal Graph Routing: Uses a temporal artifact graph to find evidence that is multiple hops away rather than relying solely on cosine similarity.