SigLIP 2 release notes / what's new
Google has released SigLIP 2, a new family of multilingual vision-language encoders that improve upon the original SigLIP by introducing additional training objectives for enhanced semantic understanding, localization, and dense feature extraction. SigLIP 2 outperforms its predecessor across all model scales in core capabilities, including image-text retrieval, zero-shot classification, and transfer performance for Vision-Language Models (VLMs).
Enhanced Training Objectives for Localization and Semantics
SigLIP 2 extends the original sigmoid loss of SigLIP with three primary technical improvements to create more structured and fine-grained visual representations.
Integration of a Text Decoder
To make the vision encoder location-aware, a text decoder is added to the training process. This decoder is tasked with three specific objectives:
- Predicting a holistic image caption.
- Predicting bounding box coordinates when provided with captions describing specific image regions.
- Predicting region-specific captions when provided with bounding box coordinates.
Self-Distillation for Local Semantics
To improve fine-grained local semantics, SigLIP 2 employs self-distillation using a teacher-student framework where the teacher is a moving average of the student's parameters. Two specific losses are introduced:
- Global-Local Loss: The student network is trained to match the teacher's representation of a full image while only seeing a partial (local) view of that image.
- Masked Prediction Loss: The student must match the teacher's features at locations where 50% of the embedded image patches have been masked.
To prevent negative effects on the encoders and reduce computational costs, these self-distillation losses are applied only after 80% of the training (which uses sigmoid and decoder loss) is complete.
Resolution Adaptation and NaFlex Variants
SigLIP 2 addresses sensitivity to aspect ratios and resolutions through two distinct methodologies:
- Fixed Resolution: Checkpoints from 95% of the training process are used to resize positional and patch embeddings, followed by continued training for the target resolution.
- Dynamic Resolution (NaFlex): Based on FlexiViT and NaViT, the
naflexvariants support inputs with different sequence lengths and native aspect ratios. This allows a single model to be used for tasks with varying requirements, such as OCR and document understanding.
Users can utilize fixed-resolution models with the standard SiglipModel class, while naflex variants require the Siglip2Model class (though this is handled automatically via the Hugging Face pipeline API).
Model Family and Performance
SigLIP 2 introduces a wider range of model sizes and configurations compared to the original SigLIP, including a new "Giant" (1B) series. The available models include:
| Size | Patch Size | Resolution |
|---|---|---|
| Base (86M) | 32 / 16 | 224 to 512 / NaFlex |
| Large (303M) | 16 | 256 to 512 |
| Shape Optimized 400M | 14 / 16 | 224 to 512 / NaFlex |
| Giant (1B) | 16 | 256 to 384 |
Evaluation data indicates that SigLIP 2 is superior to SigLIP 1 across these scales. These encoders are particularly valuable for building VLMs; for example, the PaliGemma 2 model integrates SigLIP with the Gemma 2 LLM, and SigLIP 2 provides a potential upgrade path for similar architectures.
Implementation and Inference
Inference can be performed using the transformers library. For SigLIP 2 support, users must install transformers from the main branch or the specific stable branch: pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2.
Common use cases include zero-shot image classification via the pipeline API and extracting image embeddings using AutoModel and AutoProcessor for downstream tasks.