PaliGemma Vision Language Model Release

Google has introduced PaliGemma, a family of open vision-language models (VLMs) designed to process image and text inputs to generate text outputs. These models are specifically engineered to be fine-tuned for downstream tasks rather than used as general-purpose conversational agents.

Model Architecture and Components

PaliGemma utilizes a combined architecture consisting of a vision encoder and a text decoder connected by a linear adapter:

  • Image Encoder: SigLIP-So400m, a state-of-the-art model trained jointly on images and text.
  • Text Decoder: Gemma-2B, a decoder-only model for text generation.
  • Integration: A linear adapter projects the image embeddings from SigLIP to match the 2048-dimension embeddings used by Gemma.

Model Variants and Specifications

Google has released PaliGemma in three primary checkpoint types, available in multiple resolutions and precisions:

Checkpoint Types

  • Pretrained (PT): Base models intended to be fine-tuned for specific downstream tasks.
  • Mix: Models fine-tuned on a mixture of tasks, suitable for general-purpose inference and research.
  • Fine-tuned (FT): Models specialized on specific academic benchmarks for research purposes.

Technical Configurations

  • Resolutions: Models are available in 224x224, 448x448, and 896x896. While higher resolutions can improve performance in fine-grained tasks like OCR, they significantly increase memory requirements due to longer input sequences.
  • Precisions: Checkpoints are provided in bfloat16, float16, and float32.

Core Capabilities

PaliGemma is a single-turn model that uses task prefixes (e.g., "detect" or "segment") to condition its behavior. Key capabilities demonstrated by the "mix" checkpoints include:

  • Image Captioning: Generating descriptive text for images based on prompts.
  • Visual Question Answering (VQA): Answering specific questions about the content of an image.
  • Object Detection: Identifying entities and providing bounding box coordinates. Coordinates are output as special <loc[value]> tokens (normalized to 1024), representing y_min, x_min, y_max, x_max.
  • Referring Expression Segmentation: Segmenting entities based on natural language descriptions, outputting both bounding boxes and segmentation tokens.
  • Document Understanding: Reasoning and extracting information from documents (OCR-QA).

Performance Benchmarks

Mix Checkpoints

Model MMVP Accuracy POPE Accuracy (random/popular/adversarial)
mix-224 46.00 88.00 / 86.63 / 85.67
mix-448 45.33 89.37 / 88.40 / 87.47

Fine-tuned (FT) Checkpoints

Model Name Dataset/Task Score
paligemma-3b-ft-vqav2-448 Diagram Understanding 85.64 Accuracy on VQAV2
paligemma-3b-ft-cococap-448 COCO Captions 144.6 CIDEr
paligemma-3b-ft-science-qa-448 Science QA 95.93 Accuracy on ScienceQA Img subset (no CoT)
paligemma-3b-ft-refcoco-seg-896 Object References 76.94 Mean IoU (refcoco) / 72.18 (refcoco+) / 72.22 (refcocog)
paligemma-3b-ft-rsvqa-hr-224 Remote Sensing VQA 92.61 Accuracy (test) / 90.58 Accuracy (test2)

Inference and Implementation Details

Data Processing Pipeline

  1. Text Processing: Input text is tokenized, prefixed with a <bos> token, and appended with a newline (\n) token.
  2. Image Processing: Images are resized using bicubic resampling. The SigLIP encoder generates image embeddings (1152 dimensions), which the linear projector then converts to 2048 dimensions.
  3. Sequence Construction: The model prepends a fixed number of <image> tokens based on resolution: 256 for 224 models, 1024 for 448 models, and 4096 for 896 models.
  4. Generation: The model uses full block attention for the combined input (image + bos + prompt + \n) and causal attention for the generated text.

Deployment and Fine-tuning

PaliGemma is integrated with the Hugging Face transformers library via the PaliGemmaForConditionalGeneration class. It supports 4-bit and 8-bit loading via BitsAndBytesConfig and is compatible with PEFT for LoRA and QLoRA fine-tuning. For those using the original JAX implementation, the model is available via the big_vision codebase.

Sources