zai-org/GLM-OCR

GLM-OCR: Accurate × Fast × Comprehensive

GLM‑OCR – Multimodal OCR model for complex documents

What it is – GLM‑OCR is a 0.9 B‑parameter multimodal OCR system built on the GLM‑V encoder‑decoder architecture. It combines a visual encoder (CogViT) with a lightweight cross‑modal connector and a GLM‑0.5B language decoder, plus a two‑stage pipeline (layout analysis → parallel region recognition) powered by PP‑DocLayout‑V3. The model is trained with a Multi‑Token Prediction loss and full‑task reinforcement learning to boost accuracy and generalisation on challenging documents (tables, formulas, code, seals, etc.).

Key capabilities

  • State‑of‑the‑art accuracy – 94.62 % on OmniDocBench V1.5, top rank on several document‑understanding benchmarks.
  • Compact & fast – 0.9 B parameters; can be served with vLLM, SGLang or Ollama, giving low latency and low compute cost.
  • Flexible deployment – Cloud MaaS API (no GPU), self‑hosted via vLLM/SGLang, Apple‑silicon‑optimized MLX, or Ollama. Also supports a GPU‑less client‑server mode.
  • Easy integration – Python SDK (glmocr), CLI, Flask server, and a “Skill” mode for agent‑style calls.

Installation

# Quick, cloud‑oriented use (no GPU needed)
pip install glmocr

# Self‑hosted pipeline (includes layout detector)
pip install "glmocr[selfhosted]"

# Optional Flask server
pip install "glmocr[server]"

For development you can install from source with a uv virtual‑env.

Typical usage

  • CLI – glmocr parse path/to/image.png (supports directories, custom config, layout device selection, etc.).
  • Python API –
    from glmocr import parse, GlmOcr
result = parse("image.png")
result.save("./out")
# or class‑based for more control
with GlmOcr(layout_device="cpu") as ocr:
    r = ocr.parse("image.png")
  • Flask service – python -m glmocr.server then POST JSON {"images": ["./doc.png"]}.

Deployment options

  1. Zhipu MaaS API – configure config.yaml with your API key; the SDK forwards requests, no local GPU needed.
  2. Self‑hosted vLLM – vllm serve zai-org/GLM-OCR --port 8080 … (supports speculative MTP decoding).
  3. Self‑hosted SGLang – similar launch command with speculative settings.
  4. Apple Silicon / MLX – see examples/mlx-deploy.
  5. Ollama – see examples/ollama-deploy.
  6. GPU‑less client – run the SDK server on a GPU machine and point other machines to it via api_url.

Model download – Hosted on Hugging Face and ModelScope (BF16 precision).

License – Code under Apache 2.0; the GLM‑OCR model weights under MIT; layout detector under Apache 2.0.

Resources

Related

  • Project
    apple-aiml-research/ml-fastvlm

    FastVLM is an efficient vision encoding framework that uses a hybrid encoder called FastViTHD to drastically reduce latency and token count for high-resolution images in Vision Language Models.

  • Project
    OpenSenseNova/SenseNova-U1

    SenseNova-U1 is a series of native multimodal models that unify visual understanding and generation into a single architecture, eliminating the need for separate modality adapters.

  • Project
    PaddlePaddle/ERNIE

    A family of large-scale multimodal MoE models and development toolkits providing state-of-the-art performance in text and vision-language understanding.

  • Project
    facebookresearch/seamless_communication

    A family of multimodal AI models for high-quality, expressive, and real-time translation across nearly 100 languages, supporting speech and text modalities.