google-deepmind/tips

What it solves

TIPS (Text-Image Pretraining with Spatial Awareness) provides foundational image-text encoders that improve spatial awareness in computer vision and multimodal applications. It addresses the need for vision encoders that can better align text descriptions with specific spatial regions of an image.

How it works

The project implements a series of vision-language pretraining models (TIPSv1 and TIPSv2). TIPSv2 specifically enhances patch-text alignment to improve how the model understands the relationship between text and image patches. The models are available as both PyTorch and JAX (via the Scenic library) implementations for inference.

Who it’s for

This is for researchers and developers working on general-purpose computer vision, multimodal AI, and tasks such as zero-shot segmentation, depth and normals estimation, and image-text retrieval.

Highlights

• Dual Framework Support: Provides full implementations in both PyTorch and JAX.
• Strong Spatial Awareness: Validated across 20 datasets and 9 tasks, showing performance that matches or exceeds recent vision encoders.
• Versatile Applications: Supports various downstream tasks including zero-shot segmentation and depth estimation.
• Multiple Model Sizes: Offers a range of model scales from small (S/14) to giant (g/14) to balance performance and efficiency.