PaperGuru-AI/PaperGuru-Benchmark

What it solves

PaperGuru addresses the limitation of current LLM agents in long-horizon tasks—such as multi-day software engineering, writing 200K-token literature surveys, or reproducing research papers from PDFs—where the ceiling is set by the system's ability to remember and retrieve information across sessions rather than its reasoning capability. It replaces simple RAG or agent-specific memory hacks with a formal "Lifecycle-Aware Memory" (LAM) system.

How it works

PaperGuru implements a Capital Chunk Memory (CCM) architecture that separates memory into two surfaces: compact "chunk heads" for routing and unbounded "chunk contents" for raw text. It uses a temporal artifact graph with structural edges (e.g., citations) and historical-causality edges (e.g., deprecations) to manage information.

The system follows a four-stage pipeline:

1. Search: Identifies ranked artifact heads.
2. Extract: Generates provenance-grounded "evidence cards."
3. Reason: Uses a Compose $\rightarrow$ Critique $\rightarrow$ Mutate loop to create draft segments.
4. Verify: Produces cited, provenance-checked output.

Who it’s for

It is designed for researchers and developers building long-horizon LLM agents that require high-fidelity retrieval, versioning of information, and verifiable provenance, specifically for complex tasks like academic research and code reproduction.

Highlights

• Lifecycle-Aware Memory: Satisfies four axioms including versioned content, multi-hop relevance, bounded query cost, and provenance-grounded composition.
• State-of-the-Art Performance: Outperforms baselines on PaperBench (paper-to-code reproduction) and SurveyBench (long-form survey writing).
• Proven Utility: Assisted in the creation of 10 peer-reviewed papers accepted at top-tier venues like ICML and FSE.
• Evidence-Grounded Richness: Significantly increases the inclusion of figures, tables, and executable code in generated surveys compared to existing baselines.