PaliGemma 2 Mix Release Notes

Google has released PaliGemma 2 mix, a family of instruction-tuned vision language models (VLMs) designed to demonstrate the performance capabilities of the pre-trained PaliGemma 2 checkpoints when fine-tuned on a diverse set of academic datasets. While the original PaliGemma 2 pre-trained (pt) models are intended as base models for transfer learning on specific downstream tasks, the mix variants provide a versatile baseline for a wide range of vision-language applications.

Model Architecture and Variants

PaliGemma 2 is built upon the SigLIP vision encoder and Gemma 2 language model. The mix family is available in three parameter sizes and multiple resolutions to balance performance and computational efficiency:

Parameter Count Resolution Options
3B 224x224, 448x448
10B 224x224, 448x448
28B 224x224, 448x448

Models are available for both Hugging Face Transformers and JAX frameworks.

Core Capabilities and Task Support

PaliGemma 2 mix models are capable of handling four primary categories of vision-language tasks:

  • General Vision-Language Tasks: Including visual question answering (VQA) and referring to images.
  • Document Understanding: VQA focused on infographics, charts, and diagrams.
  • Text Recognition (OCR): Text detection, captioning images containing text, and VQA on text-heavy images.
  • Localization: Object detection and image segmentation.

Prompting Strategies

While PaliGemma 2 mix supports open-ended prompts for better performance, it remains compatible with task-specific prefixes used in previous iterations.

Open-ended prompts are recommended for most tasks. However, task prefixes are still functional and follow these patterns:

  • caption {lang}: Short, COCO-like captions.
  • describe {lang}: Detailed, descriptive captions.
  • ocr: Optical character recognition.
  • answer {lang} {question}: Question answering about image content.
  • question {lang} {answer}: Question generation based on a given answer.

Mandatory Prefixes: Object detection and image segmentation require specific prefixes to function:

  • detect {object description}: Returns bounding boxes for listed objects.
  • segment {object description}; {object description}: Creates image segmentation for specified objects.

Performance Comparison and Evaluation

Evaluation of the 3B and 10B variants (at 448x448 resolution) shows that larger models generally provide more precise and detailed responses, particularly in complex reasoning and document understanding.

  • General VQA: Both 3B and 10B models accurately count objects (e.g., counting candies) and provide descriptive scene analysis.
  • Document Understanding: The 10B model demonstrates superior accuracy in resolution-sensitive tasks, correctly identifying the best variant (448px) whereas the 3B model provided a less accurate response.
  • Text Recognition: Both models exhibit strong OCR capabilities, accurately extracting dates, prices, and complex menu text from images.
  • Localization: The models can detect and segment objects based on descriptive prompts (e.g., "bird on a stick") rather than being limited to simple class labels.

Implementation and Fine-tuning

PaliGemma 2 mix can be implemented using the Hugging Face transformers library. The process involves using the PaliGemmaProcessor for input handling and PaliGemmaForConditionalGeneration for model loading and $\text{bfloat16}$ precision.

For users wishing to further customize the models, the mix checkpoints can be fine-tuned using the same methodology as the pre-trained PaliGemma 2 models. Detailed tutorials are available via the smol-vision GitHub repository.

Sources