lumpinif/deepcrawl
100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.
What it solves
Deepcrawl is designed to solve the problem of extracting high-quality, LLM-ready data from websites. It targets high-frequency agent workloads that need cleaned markdown and a structured link hierarchy to reduce token costs, context switching, and hallucinations in AI agents.
How it works
It operates as a website data context extraction platform. It processes public pages to extract three primary components: cleaned markdown of the page content, a hierarchical links tree, and metadata, all optimized for LLM digestion.
Who it’s for
Developers building AI agents that require frequent and reliable website scraping for context extraction.
Highlights
- Agent-Oriented: Specifically optimized for AI agent workloads rather than general-purpose scraping.
- LLM-Optimized: Produces cleaned markdown and metadata to minimize token usage.
- Full Platform: Includes a Nextjs Dashboard, API Workers, Auth Workers, and a Database.
- Open Source: A free and open-source alternative to Firecrawl.
Related
- Project
- Project
- Project
- Project