Finetuning olmOCR for Faithful Document Extraction

TNG has developed a fine-tuned version of the olmOCR model to automate internal document processing workflows, specifically addressing the omission of critical header and footer information. This modification transforms olmOCR from a tool primarily used for LLM training data generation into a faithful OCR engine capable of capturing all document text for business applications.

The Limitation of Pipeline-Based and Original olmOCR Systems

Traditional pipeline-based OCR engines rely on chained machine-learning components for section segmentation, table parsing, and character recognition. A fundamental flaw in this approach is the failure to flatten context into a logical reading order, or linearization, which is particularly problematic for layout-rich documents containing multi-column text, floating diagrams, headers, and footers.

While Vision Language Models (VLMs) like olmOCR address the linearization problem, the original olmOCR-7B-0225-preview was trained on the olmOCR-mix-0225 dataset, which intentionally excludes headers and footers to maintain a natural reading flow for next-token prediction. Consequently, the original model ignores extraneous information that is often critical for business use cases, such as invoice parsing.

Fine-Tuning Process and Dataset Generation

To create a faithful OCR engine, TNG utilized Qwen2.5-VL-72B-Instruct to generate a synthetic dataset of 8,000 documents that captured all relevant information, including headers and footers.

Training Configuration

  • Base Model: olmOCR-7B-0225-preview
  • Hardware: 8xH100 Nvidia node
  • Training Pipeline: Open-sourced olmOCR training pipeline
  • Hyperparameters: Default hyperparameters were used with 4 gradient accumulation steps
  • Duration: 2.5 epochs
  • Tracking: Experiment tracking was conducted via MLflow

For evaluation, TNG used a customized version of the olmOCR-mix-0225 evaluation datasets, which were also augmented with header and footer information generated by Qwen2.5-VL-72B-Instruct.

Capabilities and Performance Comparison

The fine-tuned model employs a "document anchoring" prompting strategy, which preserves born-digital content by extracting raw text blocks and position information to prompt the VLM alongside the rasterized image.

Qualitative assessments demonstrate that the fine-tuned model outperforms the original olmOCR-7B-0225-preview in several key areas:

  • Header and Footer Extraction: The model now extracts critical information located at the top and bottom of documents (e.g., invoices) that the original model previously ignored.
  • Layout Preservation: The model maintains the ability to parse complex multi-column layouts and simple tables.
  • Faithfulness: The model captures all text, including what was previously considered "extraneous" data, ensuring no critical information is missing for downstream business tasks.

TNG noted that the quality of the output can vary significantly depending on the temperature settings used during inference.

Availability

The fine-tuned model has been open-sourced and is available on Hugging Face as tngtech/olmOCR-7B-faithful.

Sources