fastcrw/crw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

What it solves

fastCRW provides a unified engine for converting web content into clean, AI-ready formats like Markdown or structured JSON. It eliminates the difficulty of scraping, crawling, and searching the web while handling complex requirements like JavaScript rendering and proxy management.

How it works

The project is implemented as a high-performance Rust binary that can be run locally or accessed via a managed Cloud API. It offers five primary operations:

  • Scrape: Converts a single URL into Markdown, HTML, or JSON.
  • Crawl: Performs bounded site crawls to collect multiple pages.
  • Map: Discovers URLs across a site without needing to scrape every page.
  • Search: Searches the web and optionally scrapes the results.
  • Extract: Pulls structured fields from one or many URLs.

It also provides a Model Context Protocol (MCP) server to integrate directly with AI coding tools like Cursor, Claude Code, and Windsurf.

Who it’s for

Developers building AI agents, RAG pipelines, and data extraction tools who need high-recall, low-latency web data acquisition without managing their own scraping infrastructure.

Highlights

  • High Performance: Outperforms competitors in truth-recall and answer accuracy on benchmarks like AA-Omniscience.
  • Flexible Deployment: Available as a local Rust binary, a managed API, or a self-hosted option.
  • Lighweight: Idles at approximately 14 MB of RAM.
  • Agent-Ready: Native MCP support for seamless integration with AI coding assistants.

Related

  • Project
    pydata/bottleneck

    Bottleneck is a library of high-performance NumPy array functions written in C, designed to speed up numerical computations and moving window operations, especially when dealing with missing data.

  • Project
    OvertureMaps/data

    Overture Maps provides open-source, cloud-native global map data partitioned by theme and stored in Parquet format for efficient, large-scale spatial analysis.

  • Project
    meta-pytorch/torchcodec

    A PyTorch-native library for decoding and encoding videos, audio, and images directly into tensors for ML training and inference.

  • Project
    a-r-j/graphein

    Graphein is a Python library that converts protein, RNA, small‑molecule, and interaction‑network data into graph objects (compatible with PyTorch Geometric, DGL, etc.) for geometric deep learning. It supports PDB/AlphaFold inputs, mesh generation, CLI batch processing, and provides ready‑made dataloaders and extensive tutorials.