xberg-io/html-to-markdown

High performance and CommonMark compliant HTML to Markdown converter. Maintained by the Kreuzberg team. Kreuzberg is a fast, polyglot document intelligence engine with a Rust core. It extracts structured data from 98+ document formats using streaming parsers and built-in OCR.

What it solves

It converts real-world, often malformed HTML into clean CommonMark or Djot without losing content. It specifically handles difficult edge cases like unclosed tags, CDATA, custom elements, nested tables, and mixed encodings that typically break simpler converters.

How it works

The project uses a Rust-based core with a tiered dispatch system that automatically selects the most efficient path for a given input:

  1. Byte Scanner: A single-pass scanner for clean HTML.
  2. DOM Walker: A tolerant walker for complex inputs.
  3. html5ever Repair: A final pass for severely malformed HTML.

This architecture ensures byte-identical output across all tiers. The core is exposed to 16 different programming languages via native bindings.

Who it’s for

Developers building web scrapers, document processing pipelines, or AI-powered tools (like RAG systems) that need to transform web content into a format LLMs can easily consume.

Highlights

  • Polyglot Support: Native bindings for 16 languages including Python, Node.js, Go, Java, and Rust.
  • High Performance: Processing speeds between 19–116 MB/s on standard corpora.
  • Metadata Extraction: Ability to parse the <head> section into structured data (Open Graph, JSON-LD, etc.).
  • Robustness: Handles GFM tables with alignment and pipe escaping.
  • AI Integration: Provides plugins for major AI coding assistants like Claude Code, Cursor, and GitHub Copilot.