Qwen2-VL release: open-source 2B/7B vision-language models and 72B API with state-of-the-art image, video, and multilingual capabilities

TL;DR

Qwen released Qwen2‑VL, a new family of vision‑language models (2B, 7B open‑source, 72B via API) that achieve state‑of‑the‑art performance on image, multi‑image, and long‑video understanding, support dozens of languages in images, and can act as a visual agent for devices such as mobiles and robots.

Overview of Qwen2‑VL

Qwen2‑VL is the latest vision‑language model series built on the Qwen2 language backbone. The release includes three model sizes:

  • Qwen2‑VL‑2B – a compact model optimized for mobile deployment.
  • Qwen2‑VL‑7B – a cost‑effective model that retains image, multi‑image, and video capabilities.
  • Qwen2‑VL‑72B – a large model offered through an API, delivering top‑tier performance that often surpasses closed‑source competitors such as GPT‑4o and Claude 3.5‑Sonnet.

All three variants share a common architecture that couples a Vision Transformer (≈600 M parameters) with the Qwen2 LLM, and they are released under the Apache 2.0 license (2B and 7B) or via a commercial API (72B).

Key Technical Advances

Naive Dynamic Resolution

Qwen2‑VL introduces Naive Dynamic Resolution which maps arbitrary image resolutions to a dynamic number of visual tokens. This eliminates the fixed‑size token bottleneck of earlier models and more closely mirrors human visual perception, allowing the model to process any image size without loss of detail.

Multimodal Rotary Position Embedding (M‑ROPE)

The M‑ROPE mechanism decomposes the original rotary embedding into three components (temporal, height, width). By doing so, the model simultaneously encodes 1‑D textual, 2‑D visual, and 3‑D video positional information, improving its ability to reason over spatial and temporal relationships.

Performance Highlights

Image & Document Understanding

  • State‑of‑the‑art on benchmarks such as MathVista, DocVQA, RealWorldQA, and MTVQA.
  • The 7B model excels in document‑centric tasks (DocVQA) and multilingual text‑image understanding (MTVQA), achieving the best reported scores among open‑source models of comparable size.
  • The 2B model, despite its small footprint, still outperforms many peers on document and video tasks.

Video Comprehension

  • Qwen2‑VL can ingest videos longer than 20 minutes, enabling high‑quality video‑based Q&A, dialogue, and content creation.
  • The 72B model shows a clear advantage over GPT‑4o and Claude 3.5‑Sonnet on video benchmarks, while the 7B and 2B models remain competitive for cost‑sensitive applications.

Multilingual Text in Images

  • Beyond English and Chinese, the models understand text in most European languages, Japanese, Korean, Arabic, Vietnamese, and other scripts embedded in images.

Demonstrated Capabilities

Enhanced Object & Handwritten Text Recognition

Qwen2‑VL can identify complex object relationships and accurately read handwritten text across multiple languages. In the provided example, it enumerated the colors and numbers of stacked boxes with perfect precision.

Visual Reasoning & Code Generation

The model solves algorithmic problems presented as screenshots, generating correct Python implementations. It also interprets charts and highly distorted aspect‑ratio images, bridging visual perception with logical reasoning.

Video Summarization & Live Interaction

Qwen2‑VL can produce detailed video descriptions, answer follow‑up questions (e.g., “What color are the astronauts’ clothes?”), and sustain a live chat flow, effectively acting as a personal video assistant.

Visual Agent & Function Calling

By coupling visual cues with function calling, Qwen2‑VL can retrieve real‑time data (flight status, weather, package tracking) based on what it sees. This capability paves the way for autonomous operation of mobiles, robots, and other devices.

Limitations

  • The model cannot extract audio from videos.
  • Knowledge is frozen at June 2023.
  • Accuracy is not guaranteed for complex instructions.
  • Weaknesses remain in counting, fine‑grained character recognition, and 3‑D spatial awareness.

Getting Started

API Access (72B)

from openai import OpenAI
import os, base64

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_path = "dog_and_girl.jpeg"
base64_image = encode_image(image_path)

client = OpenAI(api_key=os.getenv("DASHSCOPE_API_KEY"),
                base_url="https://dashscope.aliyuncs.com/compatible-mode/v1")

completion = client.chat.completions.create(
    model="qwen-vl-max-0809",
    messages=[{"role": "user",
               "content": [{"type": "text", "text": "What is this?"},
                           {"type": "image_url",
                            "image_url": {"url": "https://.../dog_and_girl.jpeg"}},
                           {"type": "image_url",
                            "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}]
              }],
    top_p=0.8, stream=True, stream_options={"include_usage": True})

for chunk in completion:
    print(chunk.model_dump_json())

Open‑Source Models (2B & 7B)

Install the latest Transformers from source to avoid the KeyError: 'qwen2_vl':

pip install git+https://github.com/huggingface/transformers

Install the visual‑utility package:

pip install qwen-vl-utils

A minimal inference script (7B) using Hugging Face Transformers:

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct", device_map="auto")
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

messages = [{"role": "user",
             "content": [{"type": "image",
                          "image": "https://.../demo.jpeg"},
                         {"type": "text", "text": "Describe this image."}]}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs,
                   padding=True, return_tensors="pt")

generated_ids = model.generate(**inputs, max_new_tokens=128)
output = processor.batch_decode(generated_ids[:, inputs.input_ids.shape[-1]:],
                                 skip_special_tokens=True)
print(output)

The repository also provides guidance for quantization (AutoGPTQ, AutoAWQ), deployment with vLLM, and fine‑tuning via Llama‑Factory.

License & Availability

  • Qwen2‑VL‑2B and Qwen2‑VL‑7B – Apache 2.0, hosted on Hugging Face and ModelScope.
  • Qwen2‑VL‑72B – Commercial API (DashScope) with usage‑based pricing.

Future Directions

Qwen’s roadmap mentions building stronger vision‑language models on upcoming language backbones and expanding to additional modalities, aiming toward an “omni model” that can process vision, audio, and other sensory data in a unified framework.

References & Resources

Sources