Hugging Face Vision Language Models 2025 Update

Vision Language Models (VLMs) have evolved from simple image-to-text systems into versatile multimodal agents capable of complex reasoning, any-to-any modality generation, and physical world interaction. The current landscape is defined by a shift toward smaller, more efficient models, the integration of Mixture-of-Experts (MoE) decoders, and the emergence of specialized capabilities like multimodal RAG and robotics control.

Emerging Model Architectures and Trends

Recent developments in VLMs have introduced several new architectural paradigms that expand the scope of multimodal AI.

Any-to-Any Models

Any-to-any models can process any input modality (image, text, audio) and generate any output modality. These models use multiple encoders to fuse embeddings into a shared representation space, which is then decoded into the desired modality. Notable examples include:

  • Qwen 2.5 Omni: Utilizes a "Thinker-Talker" architecture where the "Thinker" handles text and the "Talker" produces streaming natural speech.
  • MiniCPM-o 2.6: An 8B parameter model capable of understanding and generating content across vision, speech, and language.
  • Janus-Pro-7B: Features a decoupled visual encoding architecture to separate understanding and generation processes.
  • Chameleon (Meta): An early attempt at any-to-any capabilities, later extended by Alpha-VLLM's Lumina-mGPT to include image generation.

Multimodal Reasoning Models

Reasoning models are designed to solve complex problems through long chain-of-thought processing. Kimi-VL-A3B-Thinking is a prominent example, utilizing a MoonViT image encoder and a Mixture-of-Experts (MoE) decoder (16B total parameters, 2.8B active). It is a long chain-of-thought fine-tuned and reinforcement-learning-aligned version of the Kimi-VL base model.

Small and Efficient VLMs

There is a growing trend toward "Smol" models (typically under 2B parameters) to reduce compute costs and enable local execution on consumer devices.

  • SmolVLM2: Specifically targets video understanding at scales of 256M, 500M, and 2.2B parameters.
  • Gemma 3-4B-it: One of the smallest multimodal models featuring a 128k token context window and support for over 140 languages.
  • Qwen2.5-VL-3B-Instruct: Capable of localization, document understanding, and agentic tasks with a 32k token context length.

Mixture-of-Experts (MoE) Decoders

MoE architectures replace standard Feed-Forward Network (FFN) layers with a router that selectively activates only the most relevant "experts." This results in faster inference and quicker training convergence compared to dense models, though it requires more GPU memory. Examples include Kimi-VL, MoE-LLaVA, DeepSeek-VL2, and Llama 4.

Vision-Language-Action (VLA) Models

VLAs extend VLMs by adding action and state tokens, allowing models to control physical environments or digital UIs.

  • π0 and π0-FAST: Robotics foundation models trained across 7 platforms and 68 tasks (e.g., laundry folding, object retrieval).
  • GR00T N1 (NVIDIA): A 2B foundation model for humanoid robots that converts images and language into real-time movement control.

Specialized Multimodal Capabilities

Localization: Detection, Segmentation, and Counting

VLMs now generalize traditional computer vision tasks by outputting structured text with localization tokens.

  • PaliGemma and PaliGemma 2: Use task prefixes (e.g., "segment striped cat") to output bounding box coordinates or codebook indices for segmentation masks.
  • Molmo: Capable of pointing to instances with dots and counting objects.
  • Qwen2.5-VL: Supports detection, pointing, and counting, including UI elements.

Multimodal Safety Models

To prevent jailbreaks and harmful outputs, multimodal safety models filter both inputs and outputs.

  • ShieldGemma 2: An open safety model that evaluates images against specific content policies.
  • Llama Guard 4: A dense multimodal and multilingual safety model pruned from Llama 4 Scout.

Multimodal Retrieval Augmented Generation (RAG)

Multimodal RAG bypasses brittle PDF parsing by using multimodal retrievers to identify relevant pages directly from screenshots.

  • Document Screenshot Embedding (DSE): Uses a text and image encoder to return a single vector per passage.
  • ColBERT-like models (e.g., ColPali, ColQwen2): Use a VLM as an image encoder and an LLM as a text encoder, utilizing "MaxSim" to calculate similarity between text tokens and image patches for higher precision.

Multimodal Agents and Video Understanding

GUI and Digital Agents

VLMs are increasingly used for computer and phone use. UI-TARS-1.5 (ByteDance) and MAGMA-8B are designed for UI navigation and physical interaction. Hugging Face's smolagents library now supports VLM integration, allowing agents to either receive initial images or dynamically retrieve screenshots via callbacks for real-time GUI control.

Video Language Models

Video understanding is handled by treating videos as sequences of frames, with various strategies to manage temporal relationships:

  • LongVU (Meta): Downsamples frames using DINOv2 to remove similar frames and refines the set based on the text query.
  • Qwen2.5-VL: Uses extended multimodal RoPE to understand absolute time positions and handle dynamic FPS rates.
  • Gemma 3: Accepts video frames interleaved with timestamps in the text prompt.

Alignment and Evaluation

Preference Optimization

Direct Preference Optimization (DPO) is being applied to VLMs to align model responses with human preferences using "chosen" and "rejected" answer pairs. The RLAIF-V dataset provides over 83,000 annotated samples for this purpose, and the trl library provides the DPOTrainer for implementing this workflow.

Advanced Benchmarks

As models saturate older benchmarks, new tools have emerged to assess expert-level knowledge and reasoning:

  • MMT-Bench: Contains 31,325 multi-choice questions across 32 meta-tasks (OCR, Visual Recognition, etc.) using image, text, video, and point cloud modalities.
  • MMMU-Pro: An evolution of MMMU that increases candidate options from 4 to 10 and includes vision-only inputs and real-world simulations to mimic actual display conditions.

Sources