any4ai/AnyCrawl

AnyCrawl – Fast, Scalable Web & SERP Crawling with LLM‑Ready Extraction

What it is – AnyCrawl is an open‑source toolkit (Node.js + TypeScript) for high‑performance web crawling, site‑wide scraping, and search‑engine result (SERP) fetching. It ships with multi‑threaded / multi‑process workers, built‑in caching, and optional LLM‑powered JSON extraction, making it a ready‑made back‑end for AI agents that need up‑to‑date web data.

Key capabilities

• SERP crawling – query Google (and other engines in the future) and retrieve structured results in batches.
• Web scraping – fetch a single page with three engine choices:
  • cheerio (static HTML, fastest)
  • playwright (headless Chromium with modern JS rendering)
  • puppeteer (Chrome‑based rendering)
• Site crawling – start from a root URL and traverse links with depth, limit, and scope controls (same‑domain, same‑hostname, etc.).
• Batch jobs – submit many URLs at once, get a job ID, poll for status, and retrieve paginated results.
• LLM extraction – supply a JSON schema; AnyCrawl calls an OpenAI‑compatible LLM (e.g., Atlas Cloud) to turn raw page content into structured JSON.
• Scalability – uses multi‑threading, multi‑processing, Redis‑backed queues, and Docker‑ready images for horizontal scaling.
• Proxy support – per‑request or global proxy configuration; default high‑quality residential proxies are advertised via sponsors.
• Cache layer – optional local, S3, or Redis cache to avoid re‑downloading unchanged pages.

Typical workflow

1. Self‑host (or use the public API at https://api.anycrawl.dev).
2. Generate an API key (pnpm --filter api key:generate).
3. Call the appropriate endpoint (/v1/scrape, /v1/crawl, /v1/search, or /v1/batch/scrape) with JSON payload describing URL, engine, limits, and optional json_options for LLM extraction.
4. Receive JSON response containing raw HTML/text, optional extracted fields, and metadata (status, cache info, etc.).

Getting started

• Docs: https://docs.anycrawl.dev
• Playground: https://anycrawl.dev/playground (interactive API tester).
• Docker: docker compose up then generate a key inside the container (docker compose exec api pnpm --filter api key:generate).
• CLI: The repository provides pnpm scripts for key generation and testing.

Who might use it

• AI agents that need fresh web content (e.g., retrieval‑augmented generation pipelines).
• Data‑science teams building training datasets from the web.
• SEO / market‑research tools that scrape SERP results.
• Automation platforms that require reliable, scalable scraping with optional LLM‑based structuring.

License – MIT (permissive, commercial‑friendly).

---

Built by the Any4AI team to power the AI ecosystem with reliable, easy‑to‑integrate crawling services.