Docmatix Dataset Release

Hugging Face has released Docmatix, a massive dataset for Document Visual Question Answering (DocVQA) designed to bridge the performance gap between open-source and proprietary Vision-Language Models (VLMs). Fine-tuning the Florence-2 model on a small portion of Docmatix resulted in a relative performance increase of nearly 20% on the DocVQA benchmark.

Dataset Scale and Composition

Docmatix represents a 240x increase in scale compared to previous DocVQA datasets. While the primary dataset previously used for models like Idefics2 contained 10,000 images and 39,000 question-answer (Q/A) pairs, Docmatix provides:

  • Images: 2.4 million
  • Q/A Pairs: 9.5 million
  • Source Material: 1.3 million PDF documents

Generation Pipeline and Quality Control

Docmatix was generated using the PDFA OCR dataset, which contains 2.1 million PDFs. The technical pipeline involved the following steps:

  1. Transcription and Generation: Transcriptions from PDFA were processed by a Phi-3-small model to generate Q/A pairs.
  2. Filtering: To maintain data quality, 15% of the generated Q/A pairs were discarded as hallucinations. Filtering was performed using regular expressions to detect code and by removing answers containing the keyword "unanswerable".
  3. Image Conversion: PDFs were converted to images at a resolution of 150 dpi and uploaded to the Hugging Face Hub to reduce the resource intensity of conversion for end-users.

Prompt Optimization and Diversity

To optimize the dataset, Hugging Face performed ablation studies on the first small batch of data to refine prompts for the Phi-3 model. The optimization goals included:

  • Density: Aiming for approximately four Q/A pairs per page to avoid overlap (too many pairs) or lack of detail (too few pairs).
  • Human-like Responses: Ensuring answers were neither excessively short nor long.
  • Diversity: Prioritizing minimal repetition by guiding the Phi-3 model to ask questions based on specific information within the document (e.g., "What are the titles of John Doe?").

Performance Benchmarks

Evaluation was conducted using the Florence-2 model (700M parameters). Two versions of Florence-2 were compared: one fine-tuned on the standard DocVQA dataset and one fine-tuned on a subset of Docmatix (20% of images and 4% of Q/A pairs) followed by one epoch on DocVQA for formatting.

Dataset ANSL on DocVQA Model Size
Florence-2 fine-tuned on DocVQA 60.1 700M
Florence-2 fine-tuned on Docmatix 71.4 700M
Idefics2 74.0 8B

Results indicate that the 700M parameter Florence-2 model fine-tuned on Docmatix performed only 5% worse than the 8B parameter Idefics2 model, demonstrating the efficiency of the high-scale dataset.

Sources