A Dive into Vision-Language Models

Vision-language models (VLMs) integrate vision and language modalities to process both images and natural language text, enabling complex tasks such as zero-shot image classification, image captioning, and visual question answering. By leveraging separate or fused encoders for vision and language, these models can link visual information with textual descriptions to perform reasoning and retrieval across modalities.

Core Learning Strategies for Vision-Language Pre-training

Modern vision-language models typically utilize an image encoder, a text encoder, and a fusion strategy. While earlier models relied on hand-crafted descriptors, current research predominantly uses Transformer architectures. The following five strategies define the primary approaches to pre-training these models:

1. Contrastive Learning

Contrastive learning aligns images and text into a joint feature space by minimizing the distance between matching image-text pairs and maximizing the distance between non-matching pairs.

  • Key Models: CLIP, CLOOB, ALIGN, and DeCLIP.
  • Mechanism: These models use large datasets of {image, caption} pairs. For example, CLIP uses cosine distance to measure embedding similarity.
  • Variations: LiT (Language-Image Pre-training) fine-tunes the text encoder while keeping the image encoder frozen to improve sample efficiency.

2. PrefixLM

PrefixLM treats images as a prefix to a language model, allowing the model to predict the next token in a sequence based on visual input.

  • Mechanism: Visual Transformers (ViT) divide images into patches, which are fed sequentially. In models like SimVLM and VirTex, the encoder receives concatenated image patches and prefix text, and the decoder predicts the continuation of the text.
  • Frozen PrefixLM: To increase efficiency, models like Frozen and ClipCap keep the language model frozen and only update the image encoder to align image embeddings with the frozen LM's space.
  • Advanced Architectures: Flamingo uses Perceiver Resampler modules and new cross-attention layers to condition a frozen LM on visual data, achieving state-of-the-art few-shot learning.

3. Multi-modal Fusing with Cross Attention

This strategy fuses visual information directly into the layers of a language model decoder via cross-attention mechanisms rather than using images as prefixes.

  • Key Models: VisualGPT, VC-GPT, and Flamingo.
  • Goal: This approach balances text generation capacity and visual information, which is critical when large multi-modal datasets are unavailable. FIBER improves this by inserting cross-attention layers with gating mechanisms into both vision and language backbones.

4. Masked-Language Modeling (MLM) and Image-Text Matching (ITM)

MLM and ITM align specific image regions with corresponding text segments.

  • MLM: The model predicts masked words in a caption based on the associated image. This often requires bounding boxes or object detection models (e.g., Faster-RCNN) to generate region proposals.
  • ITM: The model predicts whether a given image and caption pair actually match.
  • Key Models: VisualBERT, FLAVA, ViLBERT, LXMERT, and BridgeTower. FLAVA specifically combines MLM, ITM, Masked-Image Modeling (MIM), and contrastive learning.

5. No Training (Optimization-based)

Some strategies bridge modalities using pre-trained unimodal models without further training.

  • MaGiC: Uses an autoregressive LM to generate captions, optimizing them via a CLIP-based "Magic score."
  • ASIF: Performs similarity-based searches in a relative representation space crafted from a small dataset of ground-truth multi-modal pairs.

Multi-modal Datasets

Pre-training Datasets

VLMs are pre-trained on massive web-scraped datasets of image-text pairs. Examples include:

  • PMD: A combination of Flickr30K, COCO, and Conceptual Captions.
  • LAION-5B: A large-scale dataset that uses CLIP to filter noise and ensure high-quality pairs.
  • COCO: Contains image instances with object labels and natural sentence descriptions.
  • Conceptual Captions & Flickr30K: Consist of web-scraped images and free-form captions.

Downstream Datasets

Fine-tuning occurs on task-specific datasets:

  • Visual Question Answering (VQA): VQA, VQA v2, NLVR2, OKVQA, TextVQA, TextCaps, and VizWiz.
  • Classification & Reasoning: Hateful Memes (multi-modal classification), SNLI-VE (visual entailment), and Winoground (compositional reasoning).

Implementation in 🤗 Transformers

Hugging Face Transformers supports a wide array of VLMs, providing tools for zero-shot classification, segmentation, and VQA.

Supported Models

  • General Purpose: CLIP, FLAVA, BridgeTower, BLIP, LiT.
  • Specialized Tasks: OWL-ViT (zero-shot object detection), CLIPSeg and GroupViT (image segmentation), and X-CLIP (zero-shot video classification).
  • Architecture Templates: The VisionEncoderDecoderModel allows users to combine any Transformer-based vision encoder (e.g., ViT, Swin) with any language decoder (e.g., GPT2, BERT).

Practical Examples

  • ViLT for VQA: Users can utilize ViltForQuestionAnswering and ViltProcessor to input an image and a question (e.g., "How many cats are there?") to receive a predicted answer.
  • CLIPSeg for Segmentation: CLIPSegForImageSegmentation enables zero-shot segmentation by providing text descriptions (e.g., "a cat") to generate binary segmentation maps for specific objects within an image.

Emerging Research Areas

Vision-language representations are expanding into specialized domains and complex physical interactions:

  • Medicine: Clinical-BERT is used for medical diagnosis and report generation, while MedFuseNet handles medical VQA.
  • Image & 3D Manipulation: StyleCLIP and DiffusionCLIP enable image manipulation; AvatarCLIP and Text2Mesh facilitate 3D shape and texture manipulation.
  • Robotics: CLIPort uses joint representations for imitation learning. Large Language Models (LLMs) are being integrated into robotics for task planning and reasoning, as seen in ProgPrompt and SayCan. The emergence of open-vocabulary detection models like OWL-ViT and GLIP is expected to further integrate multi-modal models into robotic navigation and manipulation.

Sources