Hugging Face Accelerating Document AI

Hugging Face has detailed a framework for accelerating Document AI, enabling enterprises to unlock knowledge trapped in non-digital documents like invoices, reports, and forms using open-source multimodal models. The transition from text-only models to vision-language models allows for significantly higher accuracy by incorporating visual layout and structure alongside text.

Taxonomy of Document AI Use Cases

Document AI encompasses several distinct data science tasks, ranging from basic text conversion to complex visual reasoning.

Optical Character Recognition (OCR)

OCR converts typed, handwritten, or printed text into machine-encoded text and serves as the backbone for many Document AI workflows.

  • Key Models: EasyOCR, PaddleOCR, and TrOCR (which operates on single-text line images and often pairs with a detection model like CRAFT).
  • Evaluation Metrics: Character Error Rate (CER) and word-level precision, recall, and F1.

Document Image Classification

This task involves categorizing documents (e.g., as an invoice or a letter) using image, text, or both. Multimodal models that combine visual structure and text dramatically outperform text-only models.

  • Performance Comparison: On the RVL-CDIP benchmark, a BERT-base model (text only) achieves 89% accuracy, a Document Image Transformer (DiT, vision only) reaches 92%, and multimodal models like LayoutLMv3 and Donut achieve 95%.

Document Layout Analysis

Layout analysis identifies the physical structure of a document, such as headers, tables, and text segments, typically framed as an object detection problem.

  • Key Models: LayoutLMv3 and DiT (both using Mask R-CNN as a backbone).
  • Benchmark: LayoutLMv3 achieves an overall mAP (mean average precision) of 0.951 on the PubLayNet dataset.

Document Parsing

Parsing extracts specific key-value pairs (e.g., names and totals from an invoice).

  • Evolution of Models: The LayoutLM family (v1, v2, v3) introduced a step change in performance; for example, on the FUNSD benchmark, LayoutLM can reach an F1 score of 90% compared to 60% for BERT.
  • End-to-End Approaches: Newer generative models like Donut, Pix2Struct, UDOP, and larger vision-language models (LLaVa-NeXT, Idefics2, and PaliGemma) can perform parsing in an end-to-end manner (image-to-text) without requiring a separate OCR engine.

Table Detection and Extraction

This involves identifying table locations, recognizing structure (rows, columns, cells), and performing functional analysis (recognizing keys and values).

  • Key Models: Table Transformer (a DETR-like model trained on PubTables-1M).
  • Performance: Table Transformer reports an AP of 0.966 for table detection and 0.912 for structure recognition and functional analysis on PubTables-1M.

Document Visual Question Answering (DocVQA)

DocVQA allows users to ask questions about a document image and receive a text answer.

  • Key Models: LayoutLMv3 (OCR + multimodal Transformer) achieves an ANLS score of 83.37 on the DocVQA benchmark. End-to-end models like Donut, LLaVa-NeXT, and Idefics2 remove the need for OCR.
  • Risks: DocVQA models can be prone to hallucinations (providing answers not in the document) and may inherit biases from training data.

Implementation Considerations for Enterprises

Licensing and Commercial Use

Licensing is a critical factor for enterprise adoption. Some high-performing models have restrictive licenses. For example, Microsoft's LayoutLMv2 and LayoutLMv3 checkpoints are not permitted for commercial use. Teams must evaluate licenses (e.g., MIT, Apache 2.0) before beginning data collection.

Data Preparation and Annotation

  • Quality and Scale: Performance is directly tied to image quality and the volume of training data.
  • Flexibility: Teams should test multiple OCR methodologies, including open-source (Tesseract), commercial (Cloud Vision API), or integrated (Donut).
  • Annotation Strategy: Starting with a few hundred documents to validate the approach before scaling is recommended. Tools must support bounding boxes for layout and extraction tasks.

Modeling and Evaluation

  • Fine-tuning vs. Pre-training: Hugging Face recommends starting with pre-trained open-source models and fine-tuning them, as building a pre-trained model from scratch requires millions of documents and weeks of training.
  • Image Resolution: Higher resolution images improve performance by allowing models to "see" more detail. While LayoutLMv2 downscales images to 224x224, newer models like Donut, Pix2Struct, and Idefics2 maintain the original aspect ratio and high resolution, though this increases memory requirements.
  • Metric Selection: For token classification or QA, partial match metrics are often more useful than 100% exact matches (e.g., treating "Acme" and "inside Acme" as a match).

Summary of Popular Document AI Models

Model License Primary Task/Approach
LayoutLM (v1, v2, v3) MIT / CC BY-NC-SA 4.0 Multimodal / Parsing / Classification
DiT CC BY-NC-SA 4.0 Vision-based Layout Analysis
TrOCR MIT OCR
Table Transformer MIT Table Detection & Structure
Donut MIT End-to-End (OCR-free) Parsing
Pix2Struct Apache 2.0 End-to-End / High Resolution
Idefics2 Apache 2.0 Vision-Language / End-to-End
PaliGemma PaliGemma Vision-Language / End-to-End

Sources