paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
What it solves
markdown-crawler is designed to simplify the process of converting website content into a format suitable for Large Language Models (LLMs). It solves the problem of messy HTML, which is difficult for LLMs to process and chunk, by recursively crawling a site and converting each page into a clean, structured markdown file.
How it works
The tool uses a multithreaded approach to crawl websites recursively up to a specified depth. It employs BeautifulSoup for HTML parsing and the markdownify library to convert HTML content into markdown. Users can configure the tool via a CLI or as a library in Python code, allowing them to define target content using CSS selectors, exclude specific paths, and use browser-like User-Agents to bypass basic JavaScript checks.
Who it’s for
This tool is primarily for developers building RAG (Retrieval Augmented Generation) pipelines, creating datasets for LLM fine-tuning, or building expert knowledge bases for AI agents.
Highlights
- Multithreaded Crawling: Faster data collection through parallel processing.
- RAG-Ready: Normalizes documents to make chunking by header, paragraph, or sentence easier.
- Customizable Filtering: Supports CSS selectors for target content, domain matching, and path exclusion.
- Resumable Scraping: Ability to continue scraping from where the tool left off.
- CLI and Library Support: Can be used as a standalone command-line tool or integrated into Python projects.