Open-Source OCR Models Guide – Choosing, Running, and Extending Modern Vision-Language OCR

TL;DR

Hugging Face released a guide that catalogs the latest open‑source OCR models, explains their strengths (multilingual support, layout awareness, output formats), shows how to evaluate them on benchmarks, and provides ready‑to‑use tooling for local and remote inference.

1. Modern OCR Landscape – What Models Can Do Today

1.1 Core Capabilities

  • Transcription – Handles handwritten text, multiple scripts (Latin, Arabic, Japanese), mathematical expressions, chemical formulas, and page‑number tags.
  • Complex Document Elements – Recognizes images, charts, and tables; can extract coordinates, generate captions, or convert visual data into structured formats (HTML tables, Markdown tables, JSON).
  • Output Formats – Models emit one or more of:
    • DocTag – XML‑like layout‑preserving markup used by Docling models.
    • HTML – Full structural representation suitable for digital reconstruction.
    • Markdown – Human‑readable text with optional image captions, ideal for feeding LLMs.
    • JSON – Structured snippets for tables or charts.
  • Locality Awareness – Modern OCR models embed bounding‑box “anchor” metadata, preserving reading order and reducing hallucinations.
  • Prompting – Some models (e.g., granite-docling) support task‑switching prompts like "Convert this formula to LaTeX"; others are conditioned with a fixed system prompt.

1.2 Choosing the Right Format

Use‑case Preferred Output
Digital reconstruction DocTag or HTML
LLM‑driven Q&A Markdown with captions
Programmatic analysis JSON for tables/charts

2. Cutting‑Edge Open OCR Models

2.1 Model Comparison Snapshot

Model Output(s) Notable Features Size Multilingual? Avg. OlmOCR‑Bench Score
Nanonets‑OCR2‑3B Markdown + HTML tables Captions, watermark extraction, checkboxes, flowcharts 4 B ✅ (EN, ZH, FR, AR, …) N/A
PaddleOCR‑VL Markdown, JSON, HTML Handwriting, old docs, prompt‑able, image insertion 0.9 B ✅ (109 languages) N/A
dots.ocr Markdown, JSON Grounding, image insertion, handwriting 3 B ✅ (multilingual) 79.1 ± 1.0
OlmOCR‑2 Markdown, HTML, LaTeX Grounding, batch‑optimized 8 B ❌ (English only) 82.3 ± 1.1
Granite‑Docling‑258M DocTag Prompt‑based task switching, location tokens 258 M ✅ (EN, JA, AR, ZH) N/A
DeepSeek‑OCR Markdown, HTML General visual understanding, handwriting, memory‑efficient 3 B ✅ (~100 languages) 75.4 ± 1.0
Chandra Markdown, HTML, JSON Grounding, image extraction 9 B ✅ (40+ languages) 83.1 ± 0.9
Qwen3‑VL All formats (via prompting) Ancient text, handwriting, image insertion 9 B ✅ (32 languages) N/A

Note: Scores are taken from model cards evaluated on the English‑only OlmOCR benchmark.

2.2 Evaluation Benchmarks

  • OmniDocBenchmark – Diverse document types (books, magazines, textbooks); accepts HTML/Markdown tables; uses edit‑distance and tree‑edit metrics.
  • OlmOCR‑Bench – Unit‑test style evaluation focusing on English PDFs; checks table cell relations and other layout elements.
  • CC‑OCR (Multilingual) – Only benchmark with non‑English/Chinese data; lower document quality but useful for multilingual sanity checks.

Recommendation: Test models on a small, domain‑specific dataset before committing, because benchmark coverage may not reflect your use case.

3. Cost‑Efficiency Considerations

  • Parameter counts range from <1 B (PaddleOCR‑VL) to 9 B (Chandra, Qwen3‑VL).
  • Inference cost depends heavily on optimized runtimes (vLLM, SGLang) and hardware pricing. Example: OlmOCR‑2 on an H100 ($2.69/h) costs ~US$178 per million pages.
  • Quantized variants and batch‑processing scripts can further reduce per‑page cost, making open models cheaper than many closed‑source alternatives at scale.

4. Getting Started – Running Models

4.1 Local Inference with vLLM

vllm serve nanonets/Nanonets-OCR2-3B

from openai import OpenAI
import base64
client = OpenAI(base_url="http://localhost:8000/v1")
model = "nanonets/Nanonets-OCR2-3B"

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode()

def infer(img_b64):
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}}, {"type": "text", "text": "Extract the text from the above document as if you were reading it naturally."}] }],
        temperature=0.0,
        max_tokens=15000,
    )
    return resp.choices[0].message.content

print(infer(encode_image("/path/to/doc.png")))

4.2 Transformers API Example

from transformers import AutoProcessor, AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained(
    "nanonets/Nanonets-OCR2-3B",
    torch_dtype="auto",
    device_map="auto",
    attn_implementation="flash_attention_2",
)
processor = AutoProcessor.from_pretrained("nanonets/Nanonets-OCR2-3B")

prompt = "Extract the text ..."  # see blog for full prompt
image = Image.open("doc.png")
messages = [{"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=15000, do_sample=False)
print(processor.batch_decode(output_ids, skip_special_tokens=True)[0])

4.3 Apple Silicon with MLX‑VLM

pip install -U mlx-vlm
python -m mlx_vlm.generate \
  --model ibm-granite/granite-docling-258M-mlx \
  --max-tokens 4096 \
  --temperature 0.0 \
  --prompt "Convert this chart to JSON." \
  --image chart.png

4.4 Managed Deployment via Hugging Face Inference Endpoints

  1. Open the model page (e.g., nanonets/Nanonets-OCR2-3B).
  2. Click Deploy → HF Inference Endpoints.
  3. Configure GPU size; the endpoint is ready in minutes.
  4. Use the same OpenAI‑client snippet shown above, pointing to the endpoint URL.

5. Scaling with Batch Jobs

Hugging Face Jobs together with the uv-scripts/ocr repository let you run OCR on thousands of images without owning GPUs.

hf jobs uv run --flavor l4x1 \
  https://huggingface.co/datasets/uv-scripts/ocr/raw/main/nanonets-ocr.py \
  your-input-dataset your-output-dataset \
  --max-samples 100

The script automatically handles vLLM batching and writes OCR results back as a new markdown column.

6. Beyond Pure OCR – Document AI Extensions

6.1 Visual Document Retrieval

  • Retrieve top‑k PDFs directly from a text query.
  • Combine with a VLM for multimodal RAG pipelines (see the ColPali + Qwen2 VL notebook linked in the blog).
  • Choose single‑vector (memory‑efficient) or multi‑vector (higher recall) models; most are endpoint‑ready.

6.2 Vision‑Language Models for Document QA

  • Instead of converting to text first, feed the original document image and a question to a VLM such as Qwen3‑VL.
  • This preserves layout context (tables, figures, captions) that LLM‑only pipelines may miss.

7. Open Datasets – Fuel for Future Models

  • olmOCR‑mix‑0225 (AllenAI) – Used to train >70 Hub models.
  • Synthetic pipelines like isl_synthetic_ocr.
  • VLM‑generated transcriptions filtered by heuristics.
  • Domain‑specific corrected corpora (e.g., Medical History of British India) that can be repurposed as training data.

8. Closing Thoughts

Hugging Face’s guide equips practitioners with a clear decision matrix for selecting OCR models, concrete benchmark references, cost‑analysis, and ready‑to‑run tooling both locally and in the cloud. By leveraging open‑weight models and publicly available datasets, teams can build privacy‑preserving, scalable document‑understanding pipelines without relying on proprietary services.

Further Reading

  • Vision Language Models Explained
  • Vision Language Models 2025 Update
  • PP‑OCR‑v5 Blog
  • Fine‑tuning Kosmos2.5 on Grounded OCR (notebook)
  • Fine‑tuning Florence‑2 on DocVQA (notebook)
  • SOTA OCR on‑device with Core ML and dots.ocr

Sources