DeepXiv/deepxiv_sdk

Talk to research papers like talking to authors - Python package with AI agent for arXiv papers

DeepXiv SDK – Agentic Search over Full‑text Papers and the Web

What it is – deepxiv-sdk is a Python package (with a companion CLI) that lets you ask natural‑language questions and receive answers grounded in real sources. The service retrieves full‑text arXiv papers (or cached web pages), lets an LLM read the relevant sections, and streams back an answer that includes real arXiv IDs or URLs as citations.

Why it matters – Most LLM‑powered search tools only return a list of links or synthesize from abstracts. DeepXiv adds a data layer that lets autonomous agents reason over the actual content of research papers, making the output verifiable and suitable for downstream workflows (e.g., literature reviews, automated reporting, or RAG pipelines).

Core capabilities

Feature What you get
Agentic ask deepxiv ask "…" sends a query, the service picks tools, reads sources, and streams a cited answer. Two back‑ends: arxiv (full‑text papers) and --web (Google‑cached pages).
Real citations Answers contain genuine arXiv IDs or URLs; the service never fabricates identifiers.
Progressive reading CLI helpers (search, paper --brief/--head/--section) let you retrieve candidates, skim metadata, and read only the needed sections, saving tokens.
Fine‑grained filters Author, organization, venue, year, category, citation count, date ranges, etc., can be combined with AND logic for precise retrieval.
Agent integration A Reader class and example MCP‑server wrapper show how to expose ask as a tool for autonomous agents.
Multi‑effort modes `--effort default
Streaming / JSON output Answers stream to stdout; metadata (progress, quota) goes to stderr. --json returns a single JSON object.
Rate limits Free auto‑registered token: 1 000 general calls/day (no agentic calls). Registered accounts: 10 000 general calls + 30 agentic calls/day.
Supported sources arXiv (full text), Google‑cached web pages, PubMed Central, bioRxiv/medRxiv.

Quick start (CLI)

pip install deepxiv-sdk            # install package & CLI
# register an account to get a token
deepxiv config --token YOUR_KEY
# simple question over arXiv papers
deepxiv ask "what speedup does DEER report on HumanEval"
# same query over the web
deepxiv ask "Anthropic Claude API pricing tiers" --web

The answer appears on stdout; citations are printed on stderr. Redirect > answer.md to capture only the answer.

Quick start (Python)

from deepxiv_sdk import Reader
reader = Reader()                     # reads token from config / env
res = reader.agent_search(
    "what speedup does DEER report on HumanEval",
    source="arxiv", effort="default"
)
print(res["answer"])                 # streamed answer text
print(res["sources"])                # retrieval set (list of dicts)

The SDK also offers non‑agentic helpers (search, paper, trending, etc.) that return structured JSON.

When to use it

  • Research assistants that need verifiable numbers, method details, or benchmark results from the latest papers.
  • Autonomous agents that must cite sources (e.g., a RAG pipeline that must avoid hallucinations).
  • Tool‑building where you want a simple HTTP‑based backend (Reader wraps the REST API) without handling raw arXiv downloads yourself.

Known limits & gotchas

  • Agentic calls require a registered token; the auto‑registered token cannot be used for ask.
  • Effort does not change the initial recall – raising --effort adds more reading rounds but cannot retrieve papers missed in the first retrieval pass.
  • Web backend reads only cached page bodies; uncached pages contribute only their search snippet, which is marked in the output.
  • Answer truncation – if --max-answer‑tokens is hit, the response is flagged as answer_truncated; callers should treat it as incomplete.
  • Coverage – only open‑access literature is indexed (arXiv, PMC, bioRxiv/medRxiv). Subscription‑only papers are unavailable.

License & support

Bottom line

DeepXiv SDK provides a ready‑to‑use, citation‑aware search layer that bridges the gap between LLM reasoning and the actual content of scientific papers. It is especially useful for building trustworthy research assistants or any AI system that must back its answers with verifiable sources.

Related

  • Project
    K-Dense-AI/agentic-data-scientist

    An adaptive multi-agent framework that automates complex data science workflows by separating planning from execution and utilizing a self-correcting, iterative validation loop.

  • Project
    Zleap-AI/SAG

    SAG (SQL‑Retrieval Augmented Generation) is a self‑hosted knowledge‑base app that ingests documents, extracts events + entities, and uses a novel dynamic‑hyperedge SQL retrieval model to provide fast, citation‑ready search and multi‑turn agent chat. It ships a desktop client, a Next.js web UI, a FastAPI backend, a reusable Python engine (`zleap‑sag`), and a Node CLI for mounting the knowledge base (MCP) into external agents. The system runs locally on SQLite + LanceDB (or PostgreSQL) and is MIT‑licensed.

  • Project
    tavily-ai/tavily-python

    A Python SDK for the Tavily API that enables AI agents and RAG applications to perform smart web search, content extraction, and automated research.

  • Project
    ctxrs/ctx

    ctx is a Rust‑based CLI that indexes local coding‑assistant logs (Claude, Codex, Cursor, etc.) and provides fast lexical or optional semantic search over those transcripts. It lets agents retrieve exact prior messages, tool calls, and, with the paid ctx pro add‑on, map any file/line/commit/PR back to the originating agent session (a “git blame” for AI‑generated code). All processing runs locally, preserving privacy and reducing token consumption.

  • Project
    ziwenhahaha/daily-paper-reader

    An automated AI paper discovery and reading platform that uses GitHub Actions and Pages to provide personalized research feeds and AI-powered analysis.