potamides/DeTikZify

What is DeTikZify?

DeTikZify is an open‑source multimodal language model that turns sketches or raster scientific figures into TikZ code – the LaTeX‑based language used to draw high‑quality vector graphics. In other words, you give it an image of a plot, diagram, or sketch, and it outputs a semantic, editable TikZ program that reproduces the figure. The system also supports text‑to‑TikZ generation via the companion Tik*Zero adapters.

Why it matters

Editable graphics – Unlike a PNG or PDF, TikZ code can be edited, resized, and styled without loss of quality.
Automation – Saves the tedious manual transcription of a figure into TikZ, which is common in academic writing.
Iterative refinement – Uses a Monte‑Carlo Tree Search (MCTS) inference loop to improve the generated program without extra training.
Open‑source & reproducible – Model weights, datasets, and training scripts are released on Hugging Face; the code is MIT‑licensed.

Core components

Component	Role
DeTikZify model (v2.5‑8b, v2‑8b, v1‑1b, …)	Multimodal LLM that maps an input image (or text) to TikZ source.
TikZero adapters	Lightweight modules that add zero‑shot text conditioning on top of the base model.
MCTS inference engine	Searches over possible token sequences, scoring compiled TikZ programs for correctness and visual similarity.
Web UI / CLI	Simple front‑end for interactive generation, compilation, rasterisation, and saving of TikZ files.
Dataset scripts (DaTikZ)	Tools to recreate the full DaTikZ training data (original data had to be trimmed for licensing).

Getting started

1. Install the package

# Install directly from the repo (adds optional extras for examples)
pip install 'detikzify[legacy] @ git+https://github.com/potamides/DeTikZify'

The [legacy] extra is only needed for the older v1 models; omit it if you only use v2.

2. System requirements

TeX Live 2023 (full installation) – needed to compile TikZ to PDF.
Ghostscript and Poppler – for rasterising the compiled PDFs.
A GPU with at least 8 GB VRAM for the 8‑billion‑parameter models (the 1‑b model runs on CPU‑only).

3. Quick‑start CLI

# Launch the lightweight web UI (adds helpful flags via --help)
python -m detikzify.webui --light

The UI lets you drop an image or type a caption, view the generated TikZ, and download the .tex file.

Minimal Python example (image → TikZ)

from operator import itemgetter
from detikzify.model import load
from detikzify.infer import DetikzifyPipeline

# Load the latest 8‑b model (auto‑dispatches across GPUs)
pipeline = DetikzifyPipeline(*load(
    model_name_or_path="nllg/detikzify-v2.5-8b",
    device_map="auto",
    torch_dtype="bfloat16",
))

# Provide an image URL (any raster figure or hand‑drawn sketch)
fig = pipeline.sample(image="https://w.wiki/A7Cc")

# If the program compiles, render and display it
if fig.is_rasterizable:
    fig.rasterize().show()

# Run MCTS for 10 min to explore multiple candidates
candidates = set()
for score, candidate in pipeline.simulate(image="https://w.wiki/A7Cc", timeout=600):
    candidates.add((score, candidate))

# Keep the highest‑scoring TikZ program
best = sorted(candidates, key=itemgetter(0))[-1][1]
best.save("figure.tex")

The sample call gives a single guess; simulate runs the MCTS search, returning a stream of (score, TikZFigure) tuples.

Text‑to‑TikZ with TikZero

from detikzify.model import load, load_adapter
from detikzify.infer import DetikzifyPipeline

caption = "A multi‑layer perceptron with two hidden layers."
pipeline = DetikzifyPipeline(
    *load_adapter(
        *load(
            model_name_or_path="nllg/detikzify-v2-8b",
            device_map="auto",
            torch_dtype="bfloat16",
        ),
        adapter_name_or_path="nllg/tikzero-adapter",
    )
)

fig = pipeline.sample(text=caption)
if fig.is_rasterizable:
    fig.rasterize().show()

The adapter adds a text encoder so you can describe a diagram instead of providing a picture.

Where to find the models & data

Model hub – All released checkpoints live under the nllg namespace on Hugging Face (e.g., nllg/detikzify-v2.5-8b).
Datasets – nllg/datikz-v2 and nllg/datikz-v3 contain the paired image‑TikZ data used for training. The repository also provides the DaTikZ scripts to rebuild the full dataset from arXiv sources.

Documentation & community

Paper – DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ (NeurIPS 2024 spotlight). PDF available via the OpenReview badge.
Demo – A public Hugging Face Space (nllg/DeTikZify) lets you try the model without installing anything. You can duplicate the space for a private GPU if the queue is long.
Colab notebook – Quick‑start notebook linked from the README for one‑click inference on the free tier (supports only the 1‑b model).
GitHub issues – Active discussion on installation quirks, dataset recreation, and extending the MCTS parameters.

TL;DR

DeTikZify turns pictures or textual descriptions of scientific diagrams into editable TikZ code using a large multimodal LLM and an MCTS‑based search. Install via pip, load a model from Hugging Face, and either call the Python API or the provided web UI to generate, compile, and export high‑quality vector graphics.