yzr278892/arxiv-daily-researcher

What it solves

Researchers often struggle to keep up with the overwhelming volume of new papers on ArXiv and other sources. This project provides an automated system to monitor, filter, analyze, and archive research papers based on specific research themes, reducing the manual effort required to stay current with frontier technology.

How it works

The system collects papers from ArXiv and other sources (like Nature, Science, and Hugging Face Papers), then processes them through a multi-stage pipeline:

1. Collection & Merging: It scans sources and merges duplicate papers using stable identifiers.
2. Analysis: It uses a two-tier LLM approach—a "CHEAP_LLM" for scoring, translation, and TLDRs, and a "SMART_LLM" for deep PDF analysis and trend synthesis.
3. Persistence: All candidates and processing stages are tracked in a SQLite database to allow for retries and recovery from failures.
4. Delivery: The system generates reports in Markdown or HTML format and sends notifications via channels like Email, Telegram, Slack, and DingTalk.
5. Management: A modern WebUI allows users to configure research themes, keywords, scoring strategies, and monitor token usage.

Who it’s for

Academic researchers, AI engineers, and students who need a systematic way to track specific research topics and maintain a long-term archive of relevant literature.

Highlights

• Multi-source Integration: Supports ArXiv, OpenAlex, Semantic Scholar, and various journals.
• Two-Tier LLM Strategy: Optimizes cost and quality by separating simple tasks (scoring/TLDR) from complex analysis (PDF deep-dives).
• Robust Queue Management: Uses SQLite to ensure no papers are missed and allows recovery from interrupted tasks.
• Comprehensive Reporting: Generates daily research reports, trend studies, and supplementary reports with full-text search capabilities.
• Detailed Observability: Tracks token usage (including cached inputs) and provides system health diagnostics via a WebUI.