harlan-zw/mdream

☁️ The fastest HTML to markdown convertor on GitHub. Optimized for LLMs and supports streaming.

What it solves

Mdream addresses the high token costs and inefficiency of feeding raw HTML into Large Language Models (LLMs). By converting web content into clean, optimized Markdown, it significantly reduces token usage—up to 92% in some cases—making it easier and cheaper to process entire websites or complex documents for AI applications.

How it works

The project provides high-performance conversion engines written in Rust (for native speed) and JavaScript (for portability). It supports streaming, allowing it to process large HTML inputs as they arrive. It includes a crawler that can navigate websites to generate llms.txt files, which are structured summaries designed to help AI agents understand site content. It also offers advanced features like content extraction, token-efficient cleanup, and integration with vector databases for RAG (Retrieval-Augmented Generation) pipelines.

Who it’s for

It is designed for developers building AI agents, RAG systems, or LLM-powered applications that need to ingest web data. It is particularly useful for those who need to crawl documentation, summarize websites, or build AI-discoverable content for their own projects.

Highlights

  • High Efficiency: Reduces token consumption by stripping unnecessary HTML elements, tracking parameters, and redundant links.
  • Multi-Runtime Support: Offers native Rust performance, pure JavaScript for zero-dependency environments, and WASM for edge runtimes.
  • Streaming Capabilities: Converts large HTML inputs in chunks to maintain low latency.
  • Ecosystem Integrations: Includes pre-built tools for CLI usage, Docker, GitHub Actions, Vite, and Nuxt, plus support for crawling SPA (Single Page Application) sites via Playwright.

Related

  • Project
    supermemoryai/markdowner

    A fast tool that converts website content into LLM-ready markdown data to improve the quality of AI responses.

  • Project
    paulpierre/markdown-crawler

    A multithreaded web crawler that converts website pages into markdown files to simplify document parsing and chunking for RAG and LLM fine-tuning.

  • Project
    xberg-io/html-to-markdown

    A high-performance, lossless HTML-to-Markdown converter with Rust core and bindings for 16 languages, designed to handle malformed real-world HTML.

  • Project
    fastcrw/crw

    A high-performance web scraping and crawling engine that converts URLs into clean Markdown or JSON for AI agents and RAG pipelines.

  • Project
    drmingler/smart-llm-loader

    A Python package that converts documents and scanned images into clean, structured markdown chunks using OCR and LLM-powered semantic chunking for RAG systems.