any4ai/AnyCrawl

AnyCrawl – Fast, Scalable Web & SERP Crawling with LLM‑Ready Extraction

What it is AnyCrawl is an open‑source toolkit (Node.js + TypeScript) for high‑performance web crawling, site scraping, and search‑engine result (SERP) fetching. It bundles multi‑threaded / multi‑process workers, optional Redis caching, and built‑in support for LLM‑driven structured data extraction (JSON schemas). The service can be self‑hosted or used via the public API at api.anycrawl.dev.

Core capabilities

Feature	Details
SERP crawling	Query Google (currently the only supported engine) and retrieve paginated results.
Web scraping	Fetch a single URL with three engine options: cheerio (fast static HTML), playwright, or puppeteer (full JavaScript rendering).
Site crawling	Depth‑limited, domain‑scoped traversal with configurable limits and path filters.
Batch jobs	Submit a list of URLs for asynchronous bulk scraping; poll status and retrieve results.
LLM extraction	Attach a JSON schema (json_options.schema) to a scrape request; AnyCrawl forwards page content to an OpenAI‑compatible LLM (e.g., Atlas Cloud) and returns structured JSON.
Proxy support	Per‑request proxy parameter or global ANYCRAWL_PROXY_URL; default proxy supplied, plus sponsorship links for residential proxies.
Scalability	Workers can run in multiple threads or processes; Docker images pre‑install the CloakBrowser binary for efficient browser reuse.
Caching	Optional Redis or S3‑backed cache to avoid re‑downloading unchanged pages.
API‑first	All functionality is exposed via a REST API (/v1/scrape, /v1/crawl, /v1/search, /v1/batch/scrape).

Typical use‑cases

• Data collection for LLM training or RAG – Gather raw web pages, then use the built‑in schema extraction to produce clean JSON that can be indexed.
• Search‑engine monitoring – Periodically pull Google SERP results for a keyword set.
• Large‑scale web‑scraping pipelines – Combine batch jobs with Redis caching and multi‑process workers for high throughput.
• AI agents that need up‑to‑date web content – Call the API from an autonomous agent, optionally routing through residential proxies to avoid bot blocks.

Getting started

1. Try the hosted API – Grab a free API key from the AnyCrawl dashboard and call the endpoints directly (see the README examples).
2. Self‑host – Clone the repo, run docker compose up -d (or start the api service locally).
   • Generate a key: pnpm --filter api key:generate (or via docker exec).
   • Set environment variables such as ANYCRAWL_API_AUTH_ENABLED=true and optional proxy/Redis settings.
3. Run a simple scrape:

   curl -X POST https://api.anycrawl.dev/v1/scrape \
        -H 'Content-Type: application/json' \
        -H 'Authorization: Bearer <YOUR_KEY>' \
        -d '{"url":"https://example.com","engine":"cheerio"}'
   

4. Add LLM extraction – Include a json_options.schema in the request body; configure an OpenAI‑compatible provider (e.g., Atlas Cloud) via environment variables (DEFAULT_LLM_MODEL, ATLASCLOUD_API_KEY).

Documentation & community

• Full API reference and quick‑start guide: https://docs.anycrawl.dev
• Playground for interactive testing: https://anycrawl.dev/playground
• Contributions are welcomed (see CONTRIBUTING.md).
• Backers can support the project on Open Collective.

License MIT – permissive open‑source license.

Bottom line AnyCrawl provides a ready‑to‑use, production‑grade crawling/scraping service that integrates smoothly with LLM workflows, making it a practical building block for AI‑driven data pipelines, autonomous agents, and RAG systems.