billion-token-one-task/Deepgraph

What it solves

DeepGraph addresses the difficulty of finding genuine research progress in AI and computational science by automating the process of identifying open questions and verifying them through actual experiments. It moves beyond simple LLM-generated answers by creating a loop where conclusions must be confirmed or refuted by rerunnable code or data, preventing "lucky scores" and ensuring results are auditable.

How it works

DeepGraph operates through a five-step pipeline:

1. Read: Harvests arXiv papers to extract claims, methods, and results.
2. Map: Organizes this data into an evidence graph consisting of entities, relations, and contradictions.
3. Ask: Uses SQL-based structural detectors to identify open research questions without relying on an LLM.
4. Test: Converts these questions into budgeted experiments executed on CPU or GPU.
5. Judge: Employs an "evidence ladder" to issue verdicts based on fair comparisons, which then update the search loop.

Who it’s for

It is designed for researchers who need checkable conclusions and for the JouleBeat lab to accelerate its own R&D by testing new agent harnesses and benchmarks.

Highlights

• Evidence-based: Uses an evidence graph to track contradictions and relations across thousands of papers.
• Self-Auditing: Capable of retracting its own conclusions if they are found to be based on empty model outputs.
• Automated Experimentation: Turns theoretical questions into pre-registered experiments with allocated compute budgets.
• Non-LLM Discovery: Identifies research opportunities using pure SQL detectors to avoid LLM hallucinations during the "Ask" phase.