Fine-tuning Microsoft Florence-2 for DocVQA

TL;DR

Microsoft's Florence-2 is a compact vision-language model (0.2B and 0.7B parameters) that can be effectively fine-tuned for specialized tasks like Document Visual Question Answering (DocVQA). Hugging Face demonstrated that fine-tuning the model for seven epochs on the DocVQA dataset increased the Levenshtein similarity score on the validation set from 0 to 57.0.

Florence-2 Architecture and Pre-training

Florence-2 treats all computer vision tasks as sequence-to-sequence problems, taking image and text inputs to generate text and location tokens as output. The model utilizes a DaViT vision encoder for visual embeddings and BERT for text and location embeddings, which are then processed by a standard encoder-decoder transformer architecture.

Rather than relying on architecture alone, Florence-2's performance is driven by its pre-training on the FLD-5B dataset. This automated dataset contains over 5 billion annotations—including boxes, masks, captions, and grounding—across 126 million images.

Adapting Florence-2 for Visual Question Answering (VQA)

While the original Florence-2 models support captioning, object detection, and OCR, they were not released with native VQA capabilities. Initial attempts to use unsupported prompts such as <VQA>, <vqa>, and <Visual question answering> yielded unusable results, and region-to-description prompting did not fully align with VQA requirements.

To enable VQA, the model must be fine-tuned on a specific dataset. Hugging Face used the DocVQA dataset and prepended a <DocVQA> prefix to each question to guide the model.

Fine-tuning Performance and Benchmarks

Fine-tuning Florence-2 on the DocVQA dataset resulted in a significant performance leap. Using Levenshtein's similarity as the metric, the model's validation score improved from 0 (pre-fine-tuning) to 57.0 after seven epochs of training.

Training Configurations

Experiments were conducted across different resource levels to test the model's versatility:

  • Low Resource: Using a single A100 GPU in Colab with a batch size of 6 (or a T4 GPU with a batch size of 1), with the vision encoder frozen.
  • High Resource: Fine-tuning the entire model with a batch size of 64 on a cluster of 8 H100 GPUs. This process took approximately 70 minutes.

In all configurations, a small learning rate of 1e-6 was found to be most beneficial; larger learning rates led to rapid overfitting of the training set.

Implementation Details

To implement this fine-tuning process, the AutoModelForCausalLM and AutoProcessor classes from the transformers library are used. Because Florence-2 uses custom code, trust_remote_code=True must be passed during loading.

Key Code Components

  • Dataset Class: A custom DocVQADataset class is used to prepend the <DocVQA> prefix to questions and convert images to RGB.
  • Data Collator: A collate_fn is implemented to process text and images into tensors using the processor.
  • Optimization: The training loop utilizes the AdamW optimizer and a linear learning rate scheduler.

Conclusions

Florence-2's small size (0.2B and 0.7B parameters) makes it highly suitable for deployment on edge devices or cost-effective production environments. The ability to adapt the model to new tasks via fine-tuning allows it to extend beyond its out-of-the-box capabilities to handle specialized domains like document understanding.

Sources