BLIP-2: Zero-Shot Image-to-Text Generation

BLIP-2 is a zero-shot visual-language model from Salesforce Research that enables state-of-the-art image-to-text generation by bridging frozen pre-trained vision encoders and frozen large language models (LLMs). This approach significantly reduces pre-training costs and trainable parameters compared to end-to-end multimodal pre-training.

The Q-Former Architecture

BLIP-2 utilizes a lightweight Querying Transformer (Q-Former) to bridge the modality gap between vision and language. The Q-Former is the only trainable component of the architecture; both the image encoder and the LLM remain frozen during training.

Q-Former Submodules

The Q-Former consists of two submodules that share the same self-attention layers:

  • Image Transformer: Interacts with the frozen image encoder to extract visual features. It takes learnable query embeddings as input and extracts a fixed number of output features regardless of the input image resolution.
  • Text Transformer: Functions as both a text encoder and a text decoder.

Two-Stage Pre-training

The Q-Former is trained in two distinct stages:

  1. First Stage: The image encoder is frozen, and the Q-Former is trained using three specific losses:

    • Image-text contrastive loss: Calculates pairwise similarity between query outputs and text output CLS tokens.
    • Image-grounded text generation: The text transformer uses a causal mask and attends to queries, while queries attend to each other but not to the text tokens.
    • Image-text matching loss: Queries and text interact to produce a logit indicating if the text matches the image, utilizing hard negative mining for negative examples.

  2. Second Stage: The query embeddings, which now contain relevant visual information filtered through an information bottleneck, are used as a visual prefix for the input to the LLM. This phase uses a causal LM loss for image-grounded text generation.

Model Capabilities and Implementations

BLIP-2 is designed to be flexible, allowing any combination of a visual backbone (such as ViT) and an LLM (such as OPT or Flan T5).

Supported Tasks

BLIP-2 can perform several image-to-text tasks in a zero-shot manner:

  • Image Captioning: Generating a description of an image without a text prompt by starting from the BOS (beginning-of-sequence) token.
  • Prompted Image Captioning: Continuing a provided text prompt based on the visual content of the image.
  • Visual Question Answering (VQA): Answering specific questions about an image using the prompt format: Question: {} Answer:.
  • Chat-based Prompting: Creating a conversational interface by concatenating previous question-answer pairs to build context. The context is limited to 512 tokens, matching the context length of the underlying LLMs (OPT and T5).

Integration with Hugging Face Transformers

BLIP-2 is available via the Blip2ForConditionalGeneration class in the Hugging Face Transformers library. Because it is a specialized architecture, it cannot be loaded via the standard AutoModel API and requires the explicit use of the Blip2ForConditionalGeneration class, though AutoProcessor can be used to fetch the Blip2Processor.

Sources

Related

  • Dispatch
    Zero-shot image segmentation with CLIPSeg

    Hugging Face introduces CLIPSeg, a zero-shot image segmentation model that uses CLIP embeddings to create segmentation masks from either text or image prompts without requiring category-specific training.

  • Project
    OFA-Sys/Chinese-CLIP

    A Chinese-language version of the CLIP model trained on 200 million image-text pairs for cross-modal retrieval and zero-shot image classification.

  • Dispatch
    Swift Transformers Release: Run On-Device LLMs on Apple Devices

    Hugging Face released the alpha swift-transformers Swift package, swift-chat demo app, updated exporters and transformers-to-coreml tools, and Core ML versions of Llama 2 7B and Falcon 7B, enabling developers to run LLMs on-device in Apple apps.

  • Dispatch
    SigLIP 2 release notes / what's new

    Google has released SigLIP 2, a family of multilingual vision-language encoders that outperform the original SigLIP across all scales in zero-shot classification, image-text retrieval, and VLM transfer performance.

  • Dispatch
    Hugging Face Transformers: A Beginner's Guide to Open-Source ML

    Hugging Face provides a comprehensive introductory guide to using the Transformers library and Hub to deploy and run open-source machine learning models like Microsoft's Phi-2.