spider-rs/spider

Foundational low latency web data collecting in Rust

What it solves

Spider provides a high-performance solution for crawling and scraping the web at scale. It addresses the common difficulties of web data extraction, such as managing proxies, bypassing anti-bot protections, and handling JavaScript-heavy pages without sacrificing speed.

How it works

Written in Rust, Spider is a concurrency-first engine that streams pages as they are fetched rather than batching them. It primarily uses HTTP requests for speed but can automatically launch headless Chrome for pages that require JavaScript rendering. The engine can be run locally as a library or CLI, or integrated with Spider Cloud for managed infrastructure that handles proxy rotation and unblocking automatically.

Who it’s for

It is designed for developers building data pipelines, SEO monitors, and AI browsing agents, as well as teams needing to feed web data into vector stores for LLM and RAG applications.

Highlights

  • Hybrid Rendering: Uses HTTP-first fetching and only escalates to headless Chrome when JavaScript is required.
  • Scalable Architecture: Scales from a single local script to a distributed fleet using the same API.
  • Streaming Output: Pages are streamed in real-time as they arrive.
  • Built-in Stealth: Includes integrated support for proxies, retries, rate limiting, and stealth mode to avoid detection.

Related

  • Project
    us/crw

    A high-performance web scraping and crawling engine that converts URLs into clean markdown or structured JSON for AI agents and RAG pipelines.

  • Project
    h4ckf0r0day/obscura

    A lightweight, stealthy headless browser engine written in Rust for AI agents and web scraping, serving as a high-performance, low-memory replacement for headless Chrome.

  • Project
    D4Vinci/Scrapling

    An adaptive web scraping framework that features self-healing selectors, anti-bot bypass capabilities, and a scalable spider framework for concurrent crawls.

  • Project
    SeekStorm/SeekStorm

    A high-performance Rust-based search engine providing native hybrid vector and lexical search with sub-millisecond latency for billion-scale indices.

  • Project
    dondai44423/donsetch

    A high-performance web research tool for AI agents that bypasses bot detection using native Chrome TLS and provides keyless search and token-optimized content extraction.