yzr278892/arxiv-daily-researcher

What it solves

ArXiv Daily Researcher is an automated system for monitoring, filtering, and analyzing academic papers. It solves the problem of information overload for researchers who need to stay current with the latest publications from ArXiv and other sources without manually scanning through hundreds of papers daily.

How it works

The system operates through a structured pipeline:

1. Data Acquisition: It scans ArXiv (primary source) and optional sources like Nature, Science, and Hugging Face Papers, using SQLite to deduplicate and track papers.
2. Intelligent Filtering: It uses configurable scoring strategies (Core Relevance V2, Weighted Keywords V1, or Preference Learning V1) to determine which papers are relevant based on keywords and author preferences.
3. Deep Analysis: A two-tier LLM approach is used: a "CHEAP_LLM" for initial scoring, translation, and TLDRs, and a "SMART_LLM" for deep PDF analysis and trend summaries.
4. Reporting & Notification: It generates reports in Markdown and HTML formats and sends summaries via Email, Telegram, Slack, DingTalk, or Webhooks.
5. Management: A modern WebUI allows users to manage configurations, trigger research tasks, and browse archived reports.

Who it’s for

Academic researchers, students, and AI engineers who need a systematic way to track specific research topics, perform thematic trend analysis, and maintain a long-term archive of relevant literature.

Highlights

• Dual-Mode Research: Supports both daily tracking of new papers and dedicated "Trend Research" for thematic deep-dives into specific topics over a time range.
• Robust Archiving: Uses SQLite to manage queues, handle retries for failed stages, and maintain a delivery ledger to prevent duplicate processing.
• Flexible PDF Parsing: Supports both local PyMuPDF and the MinerU API for high-quality text extraction from complex paper layouts.
• Comprehensive Management: Includes a full-featured WebUI for configuration, token usage tracking, and system diagnostics.
• Multi-Platform Notifications: Integrated support for various messaging platforms to ensure research updates are delivered instantly.