lumpinif/deepcrawl

What it solves

Deepcrawl is designed to solve the problem of extracting high-quality, LLM-ready data from websites. It targets high-frequency agent workloads that need cleaned markdown and a structured link hierarchy to reduce token costs, context switching, and hallucinations in AI agents.

How it works

It operates as a website data context extraction platform. It processes public pages to extract three primary components: cleaned markdown of the page content, a hierarchical links tree, and metadata, all optimized for LLM digestion.

Who it’s for

Developers building AI agents that require frequent and reliable website scraping for context extraction.

Highlights

• Agent-Oriented: Specifically optimized for AI agent workloads rather than general-purpose scraping.
• LLM-Optimized: Produces cleaned markdown and metadata to minimize token usage.
• Full Platform: Includes a Nextjs Dashboard, API Workers, Auth Workers, and a Database.
• Open Source: A free and open-source alternative to Firecrawl.