0xMassi/webclaw

What it solves

Most web scraping tools return raw, cluttered HTML full of navigation menus, ads, and scripts that confuse LLMs. This project converts websites into clean, structured formats like Markdown, JSON, and LLM-ready context, making it easier for AI agents and RAG (Retrieval-Augmented Generation) pipelines to process web data.

How it works

It acts as an extraction engine that can be used via a CLI, a REST API, or an MCP (Model Context Protocol) server. It fetches web pages, optionally renders JavaScript-heavy sites using a local browser, and then cleans the content by stripping unnecessary elements. It supports advanced features like CSS selectors, XPath, table-to-JSON conversion, and semantic querying to extract only the relevant information needed for an AI task.

Who it’s for

This tool is designed for developers building AI agents, RAG systems, or automated data collection pipelines who need reliable, clean, and structured web data to feed into LLMs.

Highlights

• LLM-Optimized Output: Provides specific modes to return clean Markdown or context tailored for LLM consumption.
• MCP Support: Easily integrates with AI tools like Claude Desktop, Cursor, and Windsurf via the Model Context Protocol.
• Advanced Extraction: Supports CSS selectors, XPath, and automatic table-to-JSON conversion for structured data.
• Smart Filtering: Includes features to filter by text, regex, or semantic queries to extract only the content that answers a specific question.
• Developer-Friendly: Offers a CLI, a Python SDK, Docker support, and the ability to self-host the extraction server.