harvardnlp/im2markup

What it solves

Im2markup tackles the problem of reverse-engineering an image back into the markup that produced it. Instead of relying on OCR or human transcription, the system learns to directly predict the source code — for example, the LaTeX formula that a rendered equation image came from, or the HTML of a captured web page.

How it works

The system combines two components: a convolutional neural network (CNN) that reads the text and layout structure in the image, and an attention-based neural machine translation (NMT) model that generates the markup sequence. Attention also produces a visual alignment, showing which parts of the image correspond to which tokens in the output. The pipeline includes preprocessing steps (cropping, normalizing formulas, filtering bad samples, building a vocabulary) and supports training, testing, and evaluation with BLEU, edit distance, and image-based metrics.

Who it’s for

Researchers and developers working on document understanding, automated math/LaTeX transcription, or web page structure extraction. The project provides code, a dataset (IM2LATEX-100K), and pretrained models for both LaTeX and HTML tasks.

Highlights

• Uses a CNN + attention-based NMT architecture, a general approach applicable to any image-to-markup task.
• Includes a large dataset (IM2LATEX-100K) and pretrained models for math-to-LaTeX and web-page-to-HTML.
• Provides a complete pipeline: preprocessing, training, testing, and multiple evaluation metrics (BLEU, edit distance, image exact match).
• The attention mechanism yields an interpretable alignment from markup back to the image.