Twenty Years of Pandoc: A Retrospective on the Universal Document Converter

Pandoc has evolved over two decades from a small personal project into the most popular program written in Haskell, providing a universal bridge between over 50 input formats and 76 output formats. Its success is rooted in a robust architectural decision to use an Abstract Syntax Tree (AST) rather than regular expressions, enabling $N \times M$ conversion possibilities by combining $N$ readers with $M$ writers.

The Architectural Foundation: AST and Haskell

Pandoc's reliability stems from its use of a real Abstract Syntax Tree (AST) to represent documents. Unlike early Markdown implementations that relied on sequences of regular expression transformations to convert Markdown directly to HTML, Pandoc parses input into an AST, which is then rendered into the target format. This architecture ensures that adding a new reader or writer automatically enables conversions between all existing formats.

John MacFarlane chose Haskell for the project not because it was the predetermined best tool for document conversion, but as a way to learn the language. This choice provided several critical technical advantages:

  • Algebraic Data Types: These provide a clean and ergonomic way to represent structured documents.
  • Strong Type System: The compiler catches errors during large-scale refactors, allowing the maintainer to make significant architectural changes with high confidence.
  • Purity: Haskell's pure nature prevents unintended side effects, which is essential for the --sandbox mode that guarantees readers and writers cannot access the file system.
  • Developer Ergonomics: MacFarlane notes that Haskell is better suited for expressing abstractions and helping the developer "think" compared to languages like C, Ruby, or JavaScript.

Evolution of Features and Versions

Early Releases (2006–2008)

Pandoc 0.1 launched on August 3, 2006, with approximately 3,000 lines of code. It initially supported Markdown, reStructuredText, HTML, and LaTeX. Early growth was accelerated by packaging for Debian Linux and the eventual addition of the DocBook writer and standard Markdown footnote syntax in version 0.3.

Pandoc 1 (2008–2017)

This era focused on expanding the ecosystem and improving customization:

  • Format Expansion: Added support for MediaWiki, OpenDocument (ODT), and Word docx (v1.9).
  • Customization: Version 1.4 introduced a flexible template system. Later, YAML metadata blocks and custom Lua writers were added to allow users to define ad hoc output formats.
  • JSON Filters: Introduced as a way to transform the AST between the parsing and rendering phases.
  • Commonmark: MacFarlane led the effort to create an unambiguous specification for Markdown, resulting in the Commonmark spec, which Pandoc eventually integrated.

Pandoc 2 (2017–2023)

Pandoc 2.0 introduced major architectural shifts to handle complex I/O requirements:

  • PandocMonad: A new system allowed readers and writers to perform I/O (e.g., reading included files or fetching image sizes) while maintaining a "pure" instance for testing and sandboxing.
  • Lua Filters: Replacing slower JSON filters, Lua filters run in an embedded interpreter and operate directly on the AST for significantly better performance.
  • Citation Support: A native Haskell citeproc library was written from scratch to replace the external pandoc-citeproc filter, improving speed and CSL faithfulness.

Pandoc 3 (2023–Present)

To reduce the monolithic nature of the project, Pandoc 3.0 split the software into four packages: pandoc (the library), pandoc-lua-engine, pandoc-server (HTTP API), and pandoc-cli.

Recent additions include:

  • Typst Support: Integration of a reader for the modern LaTeX competitor Typst.
  • Djot: Support for Djot, a light markup syntax designed to fix the inherent flaws of Markdown.
  • WASM Compilation: Pandoc 3.9 enabled the tool to run entirely in the browser via WebAssembly, powering the "Pandoc for the People" GUI.

Project Statistics and Impact

As of 2026, Pandoc's scale is defined by the following metrics:

  • Conversion Capability: 51 input formats and 76 output formats, totaling 3,876 distinct conversion paths.
  • Codebase: The four core packages contain 85,684 lines of Haskell code; this doubles when including primary dependencies like texmath and citeproc.
  • Community: Over 600 contributors have participated, with 7,346 GitHub issues resolved.

The Future of Document Conversion and LLMs

With the rise of Large Language Models (LLMs), the necessity of deterministic converters is being questioned. While LLMs can translate between formats, MacFarlane identifies three primary advantages of Pandoc over AI-driven conversion:

  1. Ecological Efficiency: Pandoc requires significantly less energy than an LLM for the same task.
  2. Determinism: Pandoc produces the same output for the same input every time.
  3. Reliability: Currently, rule-based conversion is more reliable than the probabilistic nature of LLMs.

However, MacFarlane acknowledges that LLMs may eventually surpass rule-based systems in interpreting human intent—specifically in edge cases where humans interpret formatting in ways that are difficult to codify into a strict specification.

Community Insights

Users and contributors highlight the practical utility and maintenance quality of the project:

"In an age of vibe-coding hype it's also so nice to see how things can be extended and snowball in usefulness when things are built correctly, by hand, from basic principles."

Practical applications cited by the community include using Pandoc to normalize binary documents (like .docx) to Markdown for Git diffing, building minimal static site generators, and stripping styling from HTML copied from Word or Google Docs to retain only semantic content.

Sources