unclecode/crawl4ai

Crawl4AI – Open‑source web crawler & scraper built for LLMs and AI agents

What it is – A Python library (also available as a Docker server and a hosted cloud service) that fetches any web page, runs a real browser when needed, and returns a clean, LLM‑friendly Markdown document. It can also extract structured data (JSON) using CSS/XPath rules or any LLM you point it at, and it supports deep crawling, adaptive crawling, and batch processing.

Why it matters – Large language models work best with concise, well‑structured text. Crawl4AI removes menus, footers, and other boilerplate, adds citation links, and can produce JSON that matches a schema you define. This makes it a handy front‑end for Retrieval‑Augmented Generation (RAG) pipelines, autonomous agents, or any data‑ingestion workflow.

How you can use it

• Library – pip install -U crawl4ai; run crawl4ai-setup once to install Playwright browsers. Then use AsyncWebCrawler (or the sync version) to get result.markdown or result.extracted_content.
• CLI – crwl <url> -o markdown or crwl … --deep‑crawl bfs --max-pages 10.
• Docker server – Spin up unclecode/crawl4ai with a token, then call its REST endpoints (/md, /html, /crawl, /screenshot, etc.) from any language.
• Crawl4AI Cloud – Get a free API key in seconds, then call https://api.crawl4ai.com/scrape (or /search, /answer, /extract) without managing browsers yourself.

Key features (as listed in the README)

• Markdown generation – clean headings, tables, code blocks, citation list; configurable filters (PruningContentFilterLXML, BM25ContentFilter, LLMContentFilter).
• Structured extraction – CSS/XPath/Regex schemas for fast, LLM‑free extraction; LLMExtractionStrategy for any LLM provider; automatic schema generation; chunking and cosine‑similarity search.
• Browser control – persistent profiles, CDP remote browsers, session handling, proxy support, stealth/undetected mode, Chromium/Firefox/WebKit.
• Crawling – BFS/DFS/best‑first strategies, resume‑state, adaptive crawling that stops once enough information is gathered, URL discovery via sitemaps or Common Crawl, full‑page scrolling for infinite‑scroll sites, screenshots/PDFs, media extraction, caching, batch‑run (arun_many).
• Self‑hosting – secure token‑protected REST API, MCP integration for Claude Code/other agents, monitoring dashboard/playground, AMD64 & ARM64 Docker images.
• Cloud‑only add‑ons – browser‑free web search (/search), direct answers (/answer), extraction without your own LLM key, automatic handling of JS‑heavy sites and bot walls, one‑line MCP registration.

Installation quick‑start

pip install -U crawl4ai
crawl4ai-setup          # installs Playwright browsers
# optional health check
crawl4ai-doctor

Or with Docker:

export CRAWL4AI_API_TOKEN=$(openssl rand -hex 32)
docker run -d -p 11235:11235 \
  -e CRAWL4AI_API_TOKEN=$CRAWL4AI_API_TOKEN \
  unclecode/crawl4ai:latest

Typical code snippet

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        res = await crawler.arun(url="https://news.ycombinator.com")
        print(res.markdown)   # clean LLM‑ready markdown

asyncio.run(main())

License – Apache 2.0 (with optional badge or text attribution).

Community – Discord, GitHub issues, contribution guide, and a free‑credit cloud tier to help fund ongoing development.