webclaw: a web extraction engine that turns websites into clean markdown and structured data for AI agents

What it solves

Web scraping tools often produce outputs that are either blocked by bot-detection or filled with irrelevant boilerplate (navigation, scripts, ads) that clutter the context window of AI agents and RAG pipelines. webclaw converts websites into clean, structured formats like Markdown or JSON, removing the noise and providing LLM-ready content.

How it works

webclaw provides a core extraction engine that processes HTML into simplified formats. It offers multiple ways to interact with the tool:

• Local Execution: A CLI and MCP server that work without an account for basic extraction.
• Hosted API: A cloud service for handling JavaScript rendering, bot-protected sites, and complex research jobs.
• SDKs: Libraries for TypeScript, Python, and Go to integrate extraction into applications.
• MCP Server: A Model Context Protocol server that allows AI agents (like Claude or Cursor) to use scraping and crawling as native tools.

Who it’s for

Developers building AI agents, RAG (Retrieval-Augmented Generation) pipelines, and researchers who need to turn messy web pages into high-quality, structured data for LLMs.

Highlights

• LLM-Optimized Formats: Specifically designed outputs for Markdown, JSON, and a compact llm format for agents.
• Agent Integration: Native support for MCP, allowing agents to scrape, crawl, and summarize pages directly.
• Crawl and Map: Ability to follow links to discover and extract entire documentation sites.
• Local-First Approach: Core extraction logic is available locally, with a hosted API for advanced needs like JS rendering.
• Brand Intelligence: Specialized tools to extract brand assets such as colors, fonts, and logos.