Scry: Programmable Internet Search and MCP Server

Scry is a programmable internet search infrastructure designed for AI agents, allowing them to execute arbitrary programs and complex queries over billions of public records rather than fetching individual web pages. By implementing the Model Context Protocol (MCP), Scry integrates directly into AI assistants like Claude, ChatGPT, and Cursor, enabling agents to compute answers across a vast corpus of internet documents.

High-Performance Programmatic Search

Scry replaces traditional keyword search with a queryable database of the public web. Instead of receiving a list of links, an agent can request specific data subsets—such as every page mentioning a specific compound but not a specific patent, grouped by domain since a certain date—and receive the computed result directly.

Core Capabilities

  • SQL-like Querying: Agents use a bounded, read-only SQL dialect to perform phrase matching, time-window filtering, joins, and recursive queries (WITH RECURSIVE).
  • Vector Operations: Scry treats embeddings as first-class values. Users can mint vectors from text, perform vector arithmetic (addition, subtraction, projection), and rank corpora using centroid-based concepts or contrast axes (e.g., measuring the "drift" of a community's stance over time).
  • Advanced Text Grammar: The scry_lex predicate supports exact phrases, exclusion, regex, fuzzy matching, and proximity search.
  • Performance: The system is optimized for scale, with some queries over tens of billions of rows returning in milliseconds. For example, a scan of all Hacker News text (15.5 billion characters) was measured at 0.86 seconds.

Data Estate and Sources

Scry maintains a "held estate" of over 416 billion rows, with approximately 161 billion rows currently queryable across 43 sources. The data is categorized into several primary domains:

Domain Key Sources Description
Social Reddit, Hacker News, Bluesky, Mastodon, Stack Exchange, LessWrong Near-census archives of social discussions and community posts.
Writing & Web Common Crawl, Wikipedia, Substack, Mailing Lists A massive index of the open web, including cleaned "distillate" layers of Common Crawl.
Scholarship OpenAlex, arXiv, PubMed, Patents, Book Passages A comprehensive graph of scholarly works, citations, and full-text academic papers.
Code GitHub Archive, Software Heritage, Package Registries Historical observations of public repositories and dependency graphs (deps.dev).
Markets Kalshi, Polymarket, Manifold Unified trade streams and market probabilities from prediction markets.
Records SEC EDGAR, FEC Transactions, Google Trends Official government filings and search trend data.

Integration and Deployment

Scry is deployed as an MCP server at https://mcp.scry.io, making it compatible with any MCP-compliant client.

Setup for AI Assistants

  • Claude: Added via Settings → Connectors → Add custom connector using the Scry URL.
  • ChatGPT: Enabled via Developer mode in Settings, then adding Scry as a plugin.
  • Developer Tools: Claude Code, Codex, and Cursor can connect via the same URL or use the HTTP API with a Bearer token for headless environments.

Benchmarks and Performance

Scry claims significant performance gains in research tasks. On Google's DeepSearchQA benchmark (900 research questions), an agent using Scry achieved a 71.8% exact-set correctness rate, surpassing the 66.1% reported for Google's Gemini Deep Research Agent on the same judge model and grading prompt.

Pricing and Access

Scry utilizes a tiered pricing model that includes a "congestion pricing" approach for agents:

  • Researcher: Free for non-commercial use (includes $5 initial credit).
  • Patron: $100/month for non-commercial use, rolling over as balance.
  • Team: Starting at $2,000/month for commercial use and custom source builds.
  • Agents: Charged at $0.05 per declared second of query time.

Community Insights and Feedback

Following its announcement on Hacker News, the community raised several technical and usability points:

"How did you scrape reddit comments? Doesn't this require expensive licensing from reddit?"

Users questioned the legality and cost of scraping high-volume sources like Reddit. Others suggested that Scry democratize its data by making datasets available via P2P torrents to reduce congestion on the querying interface. Some users found the pricing terminology (specifically "second of query time") confusing and requested a more human-readable version of the website to better understand the offering quickly.

Sources

Related

  • Project
    oxylabs/web-scraper-api

    An all-in-one web data collection API that handles proxy rotation, CAPTCHA solving, and JavaScript rendering to provide structured data from public websites.

  • Dispatch
    The Cost of AI Crawlers: Lessons from git.kernel.org

    git.kernel.org reports that AI scrapers consume approximately 20% of its total CPU capacity by inefficiently rendering HTML commits instead of using git clones, leading to an ongoing arms race of proof-of-work challenges.

  • Project
    oxylabs/perplexity-scraper

    A scraper for Perplexity AI that allows developers to send prompts and collect AI-generated responses and structured metadata in formats like JSON and Markdown.

  • Project
    makecindy/cindy

    Cindy is an open-source AI agent that runs locally on your own machine, combining multiple agent engines (harnesses like Claude Code and Codex), models, and tools into one agent that does real work in your projects. This repo contains the desktop and mobile client apps (Electron and Expo/React Native) plus shared packages, organized as a pnpm monorepo. The backend service lives in a separate repository. You can bring models via the official Cindy service, your existing Claude Code/Codex Coding Plan, your own API keys, or local models. The client is free and Apache-2.0 licensed, customizable through memory, skills, automation, MCP, and plugins, with official builds including opt-out TapDB analytics.