us/crw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

What it solves

fastCRW provides a high-performance engine for turning web content into clean markdown or structured JSON, solving the difficulty of extracting high-quality, AI-ready data from URLs. It replaces the need for multiple separate tools for searching, scraping, crawling, and extracting data from the web.

How it works

It operates as a Rust-based binary that can be run locally or accessed via a managed API. The engine supports five primary operations:

  • Scrape: Converts a single URL to markdown, HTML, screenshots, or JSON.
  • Crawl: Performs a bounded site crawl to collect pages.
  • Map: Discovers URLs within a site without scraping every page.
  • Search: Searches the web and optionally scrapes the results.
  • Extract: Produces structured fields from one or many URLs.

It integrates with AI coding tools (like Cursor, Windsurf, and Claude Code) via the Model Context Protocol (MCP), allowing agents to use it as a skill.

Who it’s for

Developers building AI agents, RAG pipelines, or data extraction tools who need a fast, low-memory footprint engine to fetch and clean web content.

Highlights

  • High Efficiency: Idles at approximately 14 MB RAM.
  • High Accuracy: Outperforms competitors like Firecrawl and Crawl4AI in truth-recall and answer accuracy benchmarks.
  • Flexible Deployment: Available as a managed API, a local Rust binary, or self-hosted.
  • Agent-Ready: Native MCP server support for seamless integration with AI coding assistants.

Related

  • Project
    0xMassi/webclaw

    A web extraction tool that converts websites into clean Markdown, JSON, and LLM-ready context for AI agents and RAG pipelines.

  • Project
    firecrawl/firecrawl

    An API for searching, scraping, and interacting with the web at scale, converting website content into clean Markdown or structured data for AI agents.

  • Project
    paulpierre/markdown-crawler

    A multithreaded web crawler that converts website pages into markdown files to simplify document parsing and chunking for RAG and LLM fine-tuning.

  • Project
    unclecode/crawl4ai

    An open-source web crawler and scraper that converts websites into clean, LLM-ready Markdown for RAG, AI agents, and data pipelines.

  • Project
    any4ai/AnyCrawl

    AnyCrawl is an open‑source, Node.js/TypeScript toolkit for fast, scalable web and SERP crawling. It offers multi‑process workers, three scraping engines (cheerio, Playwright, Puppeteer), batch jobs, Redis‑backed queues, caching, proxy support, and optional LLM‑driven JSON extraction. You can self‑host via Docker or use the public API, generate API keys, and call endpoints for single‑page scrape, full‑site crawl, SERP search, or batch scraping. Ideal for AI agents, data‑collection pipelines, and SEO automation. MIT licensed.