PaliGemma 2 Release Notes
Google has released PaliGemma 2, a new iteration of its vision language models (VLMs) that integrates the SigLIP image encoder with the latest Gemma 2 text decoder. This update expands the model family from a single 3B variant to three different sizes and introduces multiple input resolution options to better balance quality and efficiency for downstream fine-tuning.
Model Architecture and Variants
PaliGemma 2 connects a compact SigLIP image encoder to the Gemma 2 language model. The available variants are based on the Gemma 2 2B, 9B, and 27B models, resulting in PaliGemma 2 sizes of 3B, 10B, and 28B parameters (accounting for the image encoder's parameters).
To provide flexibility for different use cases, each model size supports three input resolutions:
- 224x224
- 448x448
- 896x896
All checkpoints are provided in bfloat16 precision and are distributed under the Gemma license, which permits commercial use, redistribution, and the creation of model derivatives.
Pre-training Data and Capabilities
The pre-trained (pt) models are designed for easy fine-tuning on downstream tasks. They are trained on a diverse data mixture to enable high-performance adaptation with fewer examples. The pre-training dataset includes:
- WebLI: A multilingual, web-scale image-text dataset for visual semantic understanding, object localization, and multilinguality.
- CC3M-35L: English image-alt_text pairs translated into 34 additional languages via Google Cloud Translation API.
- VQ2A: An improved question-answering dataset translated into 34 additional languages.
- OpenImages: Detection and object-aware Q&A generated from the OpenImages dataset.
- WIT: Images and texts sourced from Wikipedia.
Specialized Fine-Tuned Variants: DOCCI
Google has released specific variants fine-tuned on the DOCCI dataset (image-text caption pairs) for the 3B and 10B models at 448x448 resolution. These models demonstrate robust captioning capabilities, including text rendering, spatial relation capture, and the integration of world knowledge.
According to the technical report, PaliGemma 2 shows improved factual accuracy (measured by Non Entailment Sentences or NES) compared to previous versions and other models:
| Model | Parameters | Avg Characters | Avg Sentences | NES (Lower is Better) |
|---|---|---|---|---|
| PaliGemma | 3B | 535 | 8.9 | 34.3 |
| PaliGemma 2 (3B) | 3B | 529 | 7.7 | 28.4 |
| PaliGemma 2 (10B) | 10B | 521 | 7.5 | 20.3 |
| VILA | 7B | 871 | 8.6 | 28.6 |
| LLaVA-1.5 | 7B | 395 | 4.2 | 40.6 |
Deployment and Quantization
PaliGemma 2 is integrated with the Hugging Face transformers library (version 4.47 or later) using the PaliGemmaForConditionalGeneration and AutoProcessor APIs.
Testing on the TextVQA dataset (224x224 resolution) with a 3B fine-tuned checkpoint indicates that quantization has a minimal impact on accuracy:
- bfloat16 (No quantization): 60.04% accuracy
- 8-bit: 59.78% accuracy
- 4-bit (nf4): 58.72% accuracy
Fine-Tuning and Implementation
The API for fine-tuning PaliGemma 2 remains identical to the original PaliGemma, allowing existing code to work out of the box. The Hugging Face team demonstrated the model's efficiency by LoRA-fine-tuning a PaliGemma 2 3B model on a portion of the VQAv2 validation split in 30 minutes using three A100 (80GB) GPUs.