TIGER-AI-Lab/VLM2Vec

What it solves

VLM2Vec provides a unified framework for creating powerful, fixed-dimensional embeddings for diverse data types. It addresses the challenge of creating a single model that can handle text, images, videos, audio, visual documents, and agent-centric data (like GUI elements and tool retrieval) within a shared embedding space.

How it works

The project fine-tunes state-of-the-art Vision-Language Models (VLMs), such as Qwen2-VL, using instruction-guided contrastive training. This allows the model to produce a single embedding vector for any combination of inputs, enabling efficient retrieval and classification across different modalities.

Who it’s for

This is designed for researchers and developers building multimodal retrieval systems, cross-modal search engines, and AI agents that require a shared semantic space for diverse inputs.

Highlights

• Omni-modality Support: Covers text, image, video, audio, visual documents, and agent-centric tasks.
• MMEB Benchmark: Includes a comprehensive evaluation suite (MMEB-V3) with 190 tasks to measure performance gaps across modalities.
• OmniSET: A diagnostic tool that groups semantically equivalent instances across different modalities to analyze modality effects.
• Unified Framework: A single codebase for training and evaluating embedding models across various visual and auditory formats.