Vision Language Models Explained

Vision Language Models (VLMs) are multimodal generative models that process image and text inputs to generate text outputs, enabling tasks such as visual question answering, image captioning, and document understanding. These models allow for zero-shot generalization across diverse image types, including web pages and documents, and some can provide spatial grounding through bounding boxes or segmentation masks.

Architecture of Vision Language Models

Most prominent VLMs utilize a three-part architecture to unify image and text representations for a text decoder. This structure typically consists of:

  1. Image Encoder: Extracts features from the input image.
  2. Embedding Projector: A dense neural network that aligns the image representations with the text representations.
  3. Text Decoder: Generates the final text output based on the combined embeddings.

Different training strategies are employed depending on the model. For example, LLaVA uses a CLIP image encoder and a Vicuna text decoder; it first trains only the multimodal projector to align features using image-caption pairs, then unfreezes the text decoder for further training. In contrast, KOSMOS-2 is trained end-to-end, which is more computationally expensive. Some models, like Fuyu-8B, omit the image encoder entirely, feeding image patches directly into a projection layer before the auto-regressive decoder.

Open-Source VLM Landscape

Numerous open-source VLMs are available on the Hugging Face Hub. Key models include:

  • LLaVA 1.6 (Hermes 34B): 34B parameters, 672x672 resolution.
  • DeepSeek-VL (Base and Chat): 7B parameters, 384x384 resolution, with a chat-optimized version.
  • CogVLM (Base and Chat): 17B parameters, 490x490 resolution; the Chat version supports grounding.
  • moondream2: ~2B parameters, 378x378 resolution.
  • Qwen-VL (Base and Chat): 4B parameters, 448x448 resolution; supports zero-shot object detection.
  • KOSMOS-2: ~2B parameters, 224x224 resolution; supports grounding and zero-shot object detection.
  • Yi-VL-34B: 34B parameters, 448x448 resolution; bilingual in English and Chinese.
  • Fuyu-8B: 8B parameters, 300x300 resolution; specializes in text detection within images.

Evaluating VLM Performance

Selecting the right VLM requires utilizing specialized leaderboards and benchmarks to measure reasoning and understanding capabilities.

Leaderboards

  • Vision Arena: A continuous leaderboard based on anonymous human preference voting.
  • Open VLM Leaderboard: Ranks models based on various metrics and allows filtering by size and license.

Benchmarks

  • MMMU: A comprehensive benchmark featuring 11.5K multimodal challenges requiring college-level knowledge across disciplines like engineering and arts.
  • MMBench: Consists of 3,000 single-choice questions across 20 skills (e.g., OCR, object localization). It utilizes "CircularEval," a strategy that shuffles answer choices to ensure model consistency.
  • Domain-Specific Benchmarks: These include MathVista (mathematical reasoning), AI2D (diagram understanding), ScienceQA (science questions), and OCRBench (document understanding).

Implementation and Fine-Tuning

Inference with Transformers

VLMs can be deployed for inference using the transformers library. For instance, using LlavaNextForConditionalGeneration and LlavaNextProcessor, users can pass an image and a prompt template to generate text descriptions or answers about the image.

Fine-Tuning with TRL

TRL's SFTTrainer now includes experimental support for Vision Language Models. This allows users to perform Supervised Fine-Tuning (SFT) on models like Llava 1.5 using datasets such as llava-instruct-mix-vsft, which contains 260k image-conversation pairs. The process involves:

  1. Setting a specific chat template for the VLM.
  2. Using a custom DataCollator to combine text and image pairs.
  3. Initializing the SFTTrainer with the model, dataset, and PEFT configuration to train and push the resulting checkpoint to the Hugging Face Hub.

Sources