unclecode/crawl4ai

What it solves

Crawl4AI is an open-source web crawler and scraper designed specifically to prepare web content for Large Language Models (LLMs). It solves the problem of noisy, unstructured web data by converting pages into clean, structured Markdown that is optimized for RAG (Retrieval-Augmented Generation), AI agents, and data pipelines.

How it works

The tool uses an asynchronous browser pool (via Playwright) to fetch web pages and apply various processing strategies:

• Markdown Generation: It uses heuristic-based filtering (like the BM25 algorithm) to remove noise and generate "Fit Markdown" that focuses on core information.
• Structured Extraction: It supports both LLM-driven extraction for complex data and CSS/XPath-based extraction for fast, schema-defined JSON output.
• Browser Control: It manages browser sessions, proxies, and cookies, and can execute custom JavaScript to handle dynamic content or infinite scrolling.
• Deployment: It can be run as a Python library, via a CLI, or as a Dockerized FastAPI server with a monitoring dashboard.

Who it’s for

• AI Developers: Those building RAG systems or AI agents that need high-quality, clean web data as input.
• Data Engineers: Users needing a scalable, controllable way to extract structured data from the web without being blocked by bot detection.
• LLM Application Builders: Developers who want to convert the web into a machine-readable format without paying for proprietary scraping APIs.

Highlights

• LLM-Ready Output: Produces structured Markdown with headings, tables, and citation hints.
• Stealth Mode: Mimics real users to avoid bot detection.
• Adaptive Intelligence: Learns site patterns to explore only relevant content.
• Deep Crawling: Supports BFS strategies with crash recovery and prefetch modes for faster discovery.
• Flexible Extraction: Combines LLM-driven semantic extraction with fast CSS-based schema extraction.