Yuliang-Liu/MonkeyOCRv2

MonkeyOCRv2 Vision Encoder — A Document-Native Visual Backbone

MonkeyOCRv2 – A Visual‑Text Foundation Model for Document AI

What it is – MonkeyOCRv2 is an open‑source, document‑oriented vision encoder plus a set of downstream models for OCR‑style tasks (text detection, recognition, parsing, understanding, formula recognition, etc.). It is released as a Hugging Face/ModelScope model zoo and can be dropped into any PyTorch/Transformers pipeline.

Key components

Component Purpose Typical size
Vision Encoder (MonkeyOCRv2‑S /‑B /‑AS) Extracts image features from document images. Built on ViT‑S, ViT‑B or ViTAE‑v2‑S backbones. 28 M – 113 M params
Parsing model (‑S‑Parsing /‑B‑Parsing) Multilingual layout‑aware document parsing (tables, headings, etc.). ~0.6‑0.7 B params
Understanding model (‑S‑Und /‑B‑Und) End‑to‑end document‑question‑answering (DocVQA, InfoVQA, ChartQA, …). ~1.7‑1.8 B params

All models support 17 languages (Latin + non‑Latin scripts) and have been benchmarked on MDPBench, DocVQA, OCRBench, Union14M, etc.

How to get started

  1. Create a conda env (Python 3.11) and install the required libraries – PyTorch 2.8, transformers, accelerate, flash‑attn, vllm (for fast inference) and modelscope (optional).
  2. Download weights with the provided download_model.py script, choosing a model name (e.g., MonkeyOCRv2-B).
  3. Vision encoder – run vision/extract_feature.py (or extract_feature_vitae.py for the AS variant) to obtain image embeddings.
  4. Document parsing – start a vLLM server (parsing/serve.py) and call parsing/parse.py (CLI) or use the supplied Gradio/FastAPI demos to send PDFs or images and receive Markdown layouts.
  5. Document understanding – similarly launch understanding/serve.py and run understanding/infer.py with a natural‑language question.

Special notes

  • CPU support – as of 2026‑08‑22 the parsing service can run on CPU (see docs/cpu_support.md).
  • DFlash acceleration – vLLM 0.25.1 + CUDA 12.9+ enables up to 2× faster inference for the B‑Parsing model.
  • MonkeyDoc v2 dataset – a 113 M image‑text pair corpus (≈10 TB after download) released for pre‑training document models.

Where to find the models

Typical use‑case example (Python)

from transformers import AutoModel

# Load the vision backbone
encoder = AutoModel.from_pretrained(
    "zenosai/MonkeyOCRv2-B",
    trust_remote_code=True,
    dtype="auto",
    device_map="auto",
)

Why it matters – By providing a single, document‑native visual encoder and high‑performing multilingual parsing/understanding heads, MonkeyOCRv2 lets researchers and developers build OCR pipelines that work out‑of‑the‑box on scanned PDFs, photos, scientific papers, historical documents, and even remote‑sensing reports without stitching together separate detection, recognition, and layout models.

Related

  • Project
    Yuliang-Liu/MultimodalOCR

    A collection of benchmarks (MDPBench, OCRBench v2, and OCRBench) designed to evaluate the OCR and document parsing capabilities of Large Multimodal Models across multiple languages and real-world scenarios.

  • Dispatch
    Hugging Face Accelerating Document AI

    Hugging Face provides a comprehensive guide to using open-source multimodal models to automate document classification, parsing, and visual question answering for enterprise workflows.

  • Project
    Yuliang-Liu/MonkeyOCR

    MonkeyOCR is a document parsing model that uses a Structure-Recognition-Relation triplet paradigm to convert PDFs and images into structured Markdown, supporting English and Chinese.

  • Dispatch
    Hugging Face releases vdr-2b-multi-v1 multilingual visual document retrieval model

    Hugging Face has introduced vdr-2b-multi-v1, a multilingual embedding model for visual document retrieval that enables searching complex documents without OCR by encoding page screenshots into dense vectors.

  • Project
    PaddlePaddle/PaddleOCR

    PaddleOCR is an open‑source OCR and document‑AI toolkit that converts images/PDFs into structured, LLM‑ready JSON or Markdown. It offers multilingual scene‑text models (PP‑OCRv6), a lightweight vision‑language parser (PaddleOCR‑VL‑1.6) for tables, formulas, and ancient scripts, and a structure‑aware pipeline (PP‑StructureV3). Models run on CPU, GPU, XPU, NPU or via ONNX/TensorRT, and integrate with RAG/agent platforms like Dify and RAGFlow.