firecrawl/firecrawl

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

What it solves

Firecrawl simplifies the process of gathering data from the web for AI applications. It removes the technical hurdles of web scraping—such as managing rotating proxies, handling JavaScript-heavy pages, and dealing with rate limits—by providing a clean, LLM-ready data pipeline.

How it works

Firecrawl acts as an abstraction layer between your AI agent and the internet. You can provide a URL or a natural language prompt, and the service handles the navigation, interaction (like clicking or scrolling), and extraction. It converts raw web content into structured formats like Markdown or JSON, which are optimized for LLM token efficiency and context windows. It also offers an autonomous agent mode that can research and retrieve information without you needing to specify exact URLs.

Who it’s for

It is designed for developers building AI agents, RAG (Retrieval-Augmented Generation) systems, or any application that requires real-time, structured data from the web.

Highlights

  • LLM-Optimized Output: Automatically cleans and formats web content into Markdown or structured JSON to save tokens.
  • Autonomous Agent Mode: Allows agents to perform research and data gathering by simply describing the goal, rather than manually managing URLs.
  • Full-Stack Interaction: Supports complex browser actions like clicking, scrolling, and waiting before extracting content.
  • Broad Compatibility: Provides SDKs for Python, Node.js, Go, Java, Elixir, and Rust, with native support for MCP (Model Context Protocol) clients.

Related

  • Project
    lumpinif/deepcrawl

    An open-source website data extraction platform that converts public pages into cleaned markdown and hierarchical link trees for AI agents.

  • Project
    0xMassi/webclaw

    A web extraction tool that converts websites into clean Markdown, JSON, and LLM-ready context for use in AI agents and RAG pipelines.

  • Project
    Anakin-Inc/anakin

    An open-source web scraping API that converts websites into LLM-ready markdown or structured JSON, featuring an anti-detect browser and ML-powered proxy selection.

  • Project
    firecrawl/cli

    Firecrawl CLI is a terminal tool that connects to the Firecrawl AI‑powered web‑scraping and search service. It lets you scrape pages (markdown, HTML, JSON, screenshots, etc.), run web searches with highlights, query specialised developer or government indexes, and install “skills” that teach popular AI coding assistants (Claude Code, Cursor, OpenClaw, etc.) how to use Firecrawl automatically. Authentication is interactive or via API key, and you can point the CLI at a self‑hosted Firecrawl instance. Install with `npm i -g firecrawl-cli` and start with `firecrawl <url>` or `firecrawl search "query"`.

  • Project
    any4ai/AnyCrawl

    AnyCrawl is an open‑source, Node.js‑based toolkit for fast, scalable web and SERP crawling. It offers REST APIs for single‑page scraping, full‑site traversal, batch jobs, and Google SERP fetching, with optional JavaScript rendering via Playwright/Puppeteer. Built‑in LLM extraction lets you supply a JSON schema and receive structured data from pages using any OpenAI‑compatible model (e.g., Atlas Cloud). The system supports multi‑threaded/multi‑process workers, Redis/S3 caching, and proxy configuration, and can be run via Docker or used through the public API. MIT‑licensed and aimed at AI data pipelines and autonomous agents.