BLIP-2: Zero-Shot Image-to-Text Generation

BLIP-2 is a zero-shot visual-language model from Salesforce Research that enables state-of-the-art image-to-text generation by bridging frozen pre-trained vision encoders and frozen large language models (LLMs). This approach significantly reduces pre-training costs and trainable parameters compared to end-to-end multimodal pre-training.

The Q-Former Architecture

BLIP-2 utilizes a lightweight Querying Transformer (Q-Former) to bridge the modality gap between vision and language. The Q-Former is the only trainable component of the architecture; both the image encoder and the LLM remain frozen during training.

Q-Former Submodules

The Q-Former consists of two submodules that share the same self-attention layers:

  • Image Transformer: Interacts with the frozen image encoder to extract visual features. It takes learnable query embeddings as input and extracts a fixed number of output features regardless of the input image resolution.
  • Text Transformer: Functions as both a text encoder and a text decoder.

Two-Stage Pre-training

The Q-Former is trained in two distinct stages:

  1. First Stage: The image encoder is frozen, and the Q-Former is trained using three specific losses:

    • Image-text contrastive loss: Calculates pairwise similarity between query outputs and text output CLS tokens.
    • Image-grounded text generation: The text transformer uses a causal mask and attends to queries, while queries attend to each other but not to the text tokens.
    • Image-text matching loss: Queries and text interact to produce a logit indicating if the text matches the image, utilizing hard negative mining for negative examples.

  2. Second Stage: The query embeddings, which now contain relevant visual information filtered through an information bottleneck, are used as a visual prefix for the input to the LLM. This phase uses a causal LM loss for image-grounded text generation.

Model Capabilities and Implementations

BLIP-2 is designed to be flexible, allowing any combination of a visual backbone (such as ViT) and an LLM (such as OPT or Flan T5).

Supported Tasks

BLIP-2 can perform several image-to-text tasks in a zero-shot manner:

  • Image Captioning: Generating a description of an image without a text prompt by starting from the BOS (beginning-of-sequence) token.
  • Prompted Image Captioning: Continuing a provided text prompt based on the visual content of the image.
  • Visual Question Answering (VQA): Answering specific questions about an image using the prompt format: Question: {} Answer:.
  • Chat-based Prompting: Creating a conversational interface by concatenating previous question-answer pairs to build context. The context is limited to 512 tokens, matching the context length of the underlying LLMs (OPT and T5).

Integration with Hugging Face Transformers

BLIP-2 is available via the Blip2ForConditionalGeneration class in the Hugging Face Transformers library. Because it is a specialized architecture, it cannot be loaded via the standard AutoModel API and requires the explicit use of the Blip2ForConditionalGeneration class, though AutoProcessor can be used to fetch the Blip2Processor.

Sources