harlan-zw/mdream

What it solves

Web content is often bloated with HTML tags that consume excessive tokens when fed into Large Language Models (LLMs), increasing costs and reducing the effective context window. Mdream provides a high-performance way to convert HTML to clean, token-efficient Markdown that is specifically tuned for LLM consumption.

How it works

It converts HTML into GitHub Flavored Markdown using multiple engine options: a native Rust engine for maximum speed, a WASM engine for edge/browser runtimes, and a pure JavaScript engine for zero-dependency environments. It includes a "minimal" preset and a "clean" mode that strips redundant links, empty images, and tracking parameters to minimize token usage.

Additionally, it provides a crawler package (@mdream/crawl) that can traverse entire websites to generate llms.txt artifacts—consolidated text files designed to help AI tools discover and understand site content.

Who it’s for

• AI Engineers building RAG (Retrieval-Augmented Generation) systems who need to crawl and chunk web data for vector databases.
• Developers wanting to feed web pages directly into LLM CLIs (like Claude or GPT) with minimal token waste.
• Crawl-site owners who want to make their own websites AI-discoverable via llms.txt.

Highlights

• Extreme Performance: Up to 37x faster than Turndown and significantly faster than other Rust-based converters.
• Token Optimization: Produces up to 2x fewer tokens than competing libraries and up to 92% fewer tokens than raw HTML.
• Streaming Support: The only JavaScript HTML-to-markdown converter that supports streaming for large documents.
• Versatile Deployment: Available as a Rust crate, Node.js package, Docker image, GitHub Action, and via Browser CDN.
• LLM Artifacts: Built-in capability to generate llms.txt and llms-full.txt for website discovery.