Qwen2.5-VL Release Notes

Qwen has announced the release of Qwen2.5-VL, a flagship vision-language model that represents a significant advancement over Qwen2-VL. The model is available in three sizes—3B, 7B, and 72B—with both base and instruct versions provided via Hugging Face and ModelScope.

Key Capabilities and Features

Qwen2.5-VL introduces several high-level capabilities designed for complex visual reasoning and agentic behavior:

  • Visual Agentic Behavior: The model can act as a visual agent capable of reasoning and dynamically directing tools for computer and phone use.
  • Long Video Comprehension: Qwen2.5-VL supports videos exceeding one hour in length and can pinpoint specific events by identifying relevant video segments.
  • Structured Output Generation: The model can generate structured data (such as JSON) from scans of invoices, forms, and tables, facilitating use in commerce and finance.
  • Visual Localization: It can accurately localize objects using bounding boxes or points, providing stable JSON outputs for coordinates and attributes.
  • Advanced Visual Understanding: The model is proficient in recognizing common objects, as well as analyzing complex texts, charts, icons, graphics, and layouts.

Performance Benchmarks

Qwen2.5-VL demonstrates competitive performance across various domains, including college-level problems, mathematics, document understanding, and video analysis.

  • Flagship Model (72B-Instruct): Achieves state-of-the-art (SOTA) competitive performance and shows significant advantages in understanding diagrams and documents. It is capable of functioning as a visual agent without requiring task-specific finetuning.
  • Mid-size Model (7B-Instruct): Outperforms GPT-4o-mini in several tasks.
  • Edge AI Solution (3B): Outperforms the 7B model of the previous Qwen2-VL version.

Technical Deep Dive: Model Capabilities

Document Parsing and Text Recognition

Qwen2.5-VL features upgraded OCR capabilities with improved performance in multi-language, multi-orientation, and multi-scenario text recognition. A key innovation is the QwenVL HTML format, a unique document parsing format that extracts layout information based on HTML, allowing the model to parse magazines, research papers, web pages, and mobile screenshots.

Object Grounding and Image Recognition

The model utilizes bounding boxes and point-based representations for grounding, enabling hierarchical positioning. Its image recognition has been expanded to include a vast array of categories, including plants, animals, landmarks, film/TV IPs, and various commercial products.

Video Processing

To enhance temporal processing, Qwen2.5-VL implements dynamic frame rate (FPS) training and absolute time encoding. This allows the model to achieve second-level event localization and summarize key points from hourly-scale videos.

Architectural Improvements

Compared to Qwen2-VL, Qwen2.5-VL optimizes how the model perceives spatial and temporal scales and streamlines the visual encoder for better efficiency.

Spatial and Temporal Perception

  • Spatial: The model dynamically converts images of different sizes into tokens of varying lengths. It represents coordinates (detection boxes and points) using the actual size scale of the image rather than traditional coordinate normalization, allowing the model to learn image scale directly.
  • Temporal: Dynamic FPS training and absolute time encoding align mRoPE ids with the speed of time, enabling the model to learn the pace of time through temporal dimension id intervals.

Visual Encoder Optimization

Qwen has trained a native dynamic resolution ViT from scratch. To reduce computational load and address load imbalance, the model employs Window Attention (with a maximum window size of 8x8) for most layers, reserving only four layers for Full Attention. The ViT architecture has also been updated to use RMSNorm and SwiGLU structures to maintain consistency with LLMs.

Sources