xberg-io/html-to-markdown

High performance and CommonMark compliant HTML to Markdown converter. Maintained by the Kreuzberg team. Kreuzberg is a fast, polyglot document intelligence engine with a Rust core. It extracts structured data from 98+ document formats using streaming parsers and built-in OCR.

What it solves

It converts messy, real-world HTML into clean Markdown (CommonMark or Djot) without losing content. It is designed to handle malformed input such as unclosed tags, CDATA, custom elements, broken entities, and nested tables, which often break standard parsers.

How it works

The project uses a Rust core with a tiered dispatch system that moves from a byte scanner to a DOM walker and finally to html5ever repair for the most difficult cases. This architecture ensures consistent, byte-equal output across all tiers. It provides a single convert() entry point and offers native bindings for 16 different programming languages.

Who it’s for

Developers who need to process web content for LLM data pipelines, AI coding assistants, or general content migration, and who require high performance and robust handling of broken HTML.

Highlights

  • Polyglot Support: Native bindings for 16 languages including Python, Node.js, Go, Java, C#, and Rust.
  • Metadata Extraction: Automatically parses the <head> section to extract structured metadata like Open Graph, Twitter, JSON-LD, and RDFa.
  • High Performance: Processing speeds between 19–116 MB/s on the Wikipedia/mdream corpus.
  • Robustness: Specifically handles complex HTML issues like nested tables and mixed encodings automatically.
  • AI Integration: Provides plugins for major AI coding agents like Claude Code, Cursor, and GitHub Copilot CLI.

Related

  • Project
    harlan-zw/mdream

    A high-performance HTML-to-Markdown converter optimized for LLMs, reducing token usage and supporting streaming and site-wide crawling for RAG systems.

  • Project
    firecrawl/anydoc

    anydoc is a pure‑Rust library (with Node, Python, and WebAssembly bindings) that converts 14 office‑document formats into consistent GitHub‑Flavored Markdown in a few milliseconds. It preserves document structure, supports optional hosted OCR for scanned PDFs, and is packaged as a CLI, an Agent Skill, and language‑specific SDKs, making it ideal for LLM‑oriented pipelines that need fast, uniform text extraction.

  • Project
    paulpierre/markdown-crawler

    A multithreaded web crawler that converts website pages into markdown files to simplify document parsing and chunking for RAG and LLM fine-tuning.

  • Project
    us/crw

    A high-performance web scraping and crawling engine that converts URLs into clean markdown or structured JSON for AI agents and RAG pipelines.

  • Project
    datalab-to/marker

    A high-accuracy document conversion tool that transforms PDFs and other files into structured Markdown, JSON, or HTML, specializing in tables and equations.