0xMassi/webclaw

What it solves

Web scraping tools often produce outputs that are either blocked by bot-protection or cluttered with raw HTML, navigation menus, scripts, and ads. webclaw solves this by converting websites into clean, structured content (Markdown, JSON, or LLM-optimized text) that is immediately usable for AI agents and RAG pipelines.

How it works

webclaw provides an extraction engine that strips away boilerplate and preserves structure. It can be used as a CLI tool, an MCP (Model Context Protocol) server for AI agents like Claude and Cursor, or via SDKs and a REST API. It supports local extraction for core paths and a hosted API for advanced needs like JavaScript rendering and bypassing bot protection.

Who it’s for

Developers building AI agents, RAG (Retrieval-Augmented Generation) systems, and automation workflows who need high-quality, noise-free web data extracted from URLs or entire documentation sites.

Highlights

• Multiple Output Formats: Supports Markdown, LLM-optimized text, JSON, and plain text.
• Crawl and Map: Ability to follow links to crawl entire sites or map URLs without full extraction.
• MCP Integration: Native support for AI agents via an MCP server, allowing agents to scrape, crawl, and summarize pages directly.
• Local-First Approach: Core extraction works locally without an API key, with a hosted option for complex sites.
• Brand Intelligence: Specialized tools to extract brand assets like colors, fonts, and logos.
• Broad Integration: Provides SDKs for TypeScript, Python, and Go.