TIGER-AI-Lab/VLM2Vec

This repo contains the code for "VLM2Vec / MMEB" [ICLR 2025], "VLM2Vec-V2 / MMEB-V2" [TMLR 2026], and "MMEB-V3" [COLM 2026]

What it solves

VLM2Vec provides a unified framework for creating powerful, fixed-dimensional embeddings for diverse data types. It addresses the challenge of creating a single model that can handle text, images, videos, audio, visual documents, and agent-centric data (like GUI elements and tool retrieval) within a shared embedding space.

How it works

The project fine-tunes state-of-the-art Vision-Language Models (VLMs), such as Qwen2-VL, using instruction-guided contrastive training. This allows the model to produce a single embedding vector for any combination of inputs, enabling efficient retrieval and classification across different modalities.

Who it’s for

This is designed for researchers and developers building multimodal retrieval systems, cross-modal search engines, and AI agents that require a shared semantic space for diverse inputs.

Highlights

  • Omni-modality Support: Covers text, image, video, audio, visual documents, and agent-centric tasks.
  • MMEB Benchmark: Includes a comprehensive evaluation suite (MMEB-V3) with 190 tasks to measure performance gaps across modalities.
  • OmniSET: A diagnostic tool that groups semantically equivalent instances across different modalities to analyze modality effects.
  • Unified Framework: A single codebase for training and evaluating embedding models across various visual and auditory formats.

Related

  • Project
    Tencent/WeMM-Embedding

    WeMM‑Embedding is Tencent’s open‑source multimodal embedding suite (2B/4B/9B parameters) that converts text, images, video, and visual documents into a single L2‑normalised vector. It supports a range of output dimensions via “Matryoshka” truncation, offers ready‑made inference scripts for 🤗 Transformers and Sentence‑Transformers, and includes serving wrappers for vLLM and SGLang. Benchmarks (MMEB‑v2/v3) show state‑of‑the‑art scores across image, video, document, text, and agent tasks. Models are hosted on Hugging Face and released under Apache 2.0.

  • Project
    facebookresearch/VLM3

    VLM³ is a framework that enables standard Vision Language Models to perform 3D vision tasks like depth estimation and camera pose estimation using only text-based outputs and data scaling.

  • Project
    bytedance/Sa2VA

    Sa2VA is a ByteDance research codebase that fuses SAM‑2 segmentation with multimodal LLMs (InternVL, Qwen‑VL) to provide pixel‑level grounded understanding for images and videos. It includes the core Sa2VA model, the VRT reasoning benchmark, the SAMTok mask‑token interface, and extensions like SaSaSa2VA. The repo uses the `uv` package manager for reproducible environments, provides setup scripts, and links to papers, model zoo, and datasets.

  • Project
    Osilly/Vision-DeepResearch

    A framework and benchmark for developing Multimodal Large Language Models capable of deep research through iterative visual and textual search across images and videos.

  • Project
    vllm-project/vllm-omni

    A framework that extends vLLM to provide high-performance serving for omni-modality models, supporting text, image, audio, video, and action outputs across various hardware backends.