TIGER-AI-Lab/VLM2Vec

This repo contains the code for "VLM2Vec / MMEB" [ICLR 2025], "VLM2Vec-V2 / MMEB-V2" [TMLR 2026], and "MMEB-V3" [COLM 2026]

What it solves

VLM2Vec provides a unified framework for creating powerful, fixed-dimensional embeddings for diverse data types. It addresses the challenge of creating a single model that can handle text, images, videos, audio, visual documents, and agent-centric data (like GUI elements and tool retrieval) within a shared embedding space.

How it works

The project fine-tunes state-of-the-art Vision-Language Models (VLMs), such as Qwen2-VL, using instruction-guided contrastive training. This allows the model to produce a single embedding vector for any combination of inputs, enabling efficient retrieval and classification across different modalities.

Who it’s for

This is designed for researchers and developers building multimodal retrieval systems, cross-modal search engines, and AI agents that require a shared semantic space for diverse inputs.

Highlights

  • Omni-modality Support: Covers text, image, video, audio, visual documents, and agent-centric tasks.
  • MMEB Benchmark: Includes a comprehensive evaluation suite (MMEB-V3) with 190 tasks to measure performance gaps across modalities.
  • OmniSET: A diagnostic tool that groups semantically equivalent instances across different modalities to analyze modality effects.
  • Unified Framework: A single codebase for training and evaluating embedding models across various visual and auditory formats.

Related

  • Project
    Tencent/WeMM-Embedding

    WeMM-Embedding is a family of multimodal models that convert text, images, videos, and visual documents into unified numerical representations for cross-modal search and analysis.

  • Project
    facebookresearch/VLM3

    VLM³ is a framework that enables standard Vision Language Models to perform complex 3D vision tasks like depth estimation and camera pose estimation without needing specialized architectures or data augmentations.

  • Project
    bytedance/Sa2VA

    Sa2VA is a ByteDance research codebase that fuses SAM‑2 segmentation with multimodal LLMs (InternVL, Qwen‑VL) to provide pixel‑level grounded understanding for images and videos. It includes the core Sa2VA model, the VRT reasoning benchmark, the SAMTok mask‑token interface, and extensions like SaSaSa2VA. The repo uses the `uv` package manager for reproducible environments, provides setup scripts, and links to papers, model zoo, and datasets.

  • Project
    Osilly/Vision-DeepResearch

    A framework and benchmark for developing Multimodal Large Language Models capable of deep research through iterative visual and textual search across images and videos.

  • Project
    vllm-project/vllm-omni

    vLLM‑Omni is an open‑source extension of the vLLM inference engine that adds fast, scalable serving for multimodal models (text, image, audio, video, and action). It supports both autoregressive and diffusion‑style architectures, provides a full‑duplex realtime API, works across GPUs, NPUs, and other accelerators, and ships ready‑to‑run recipes for many popular Hugging Face models. The project is production‑grade, Apache‑2.0 licensed, and backed by documentation, a Slack community, and recent technical reports.