Hugging Face releases vdr-2b-multi-v1 multilingual visual document retrieval model

Hugging Face has introduced vdr-2b-multi-v1, a multilingual embedding model designed for visual document retrieval that encodes document page screenshots into dense single-vector representations. This approach allows users to search and query visually rich documents without the need for OCR, data extraction pipelines, or manual chunking.

Key Model Capabilities

vdr-2b-multi-v1 provides several technical advantages over previous visual retrieval methods:

  • Multilingual Support: The model is trained on Italian, Spanish, English, French, and German, enabling cross-lingual retrieval (e.g., searching for German documents using Italian queries).
  • Efficiency and Speed: The English-only version, vdr-2b-v1, achieves 3x faster inference and significantly lower VRAM usage compared to base models by using 768 image patches instead of 2560.
  • Matryoshka Representation Learning (MRL): The model supports variable vector sizes; users can reduce vector dimensions by 3x while retaining 98% of embedding quality, optimizing for storage and retrieval speed.
  • OCR-Free Retrieval: By encoding screenshots directly, the model bypasses the complexities of traditional text-extraction pipelines.

The vdr-multilingual-train Dataset

To address the scarcity of multilingual multimodal datasets, Hugging Face released the vdr-multilingual-train dataset. This is the largest open-source multilingual synthetic dataset for visual document retrieval, containing 500,000 high-quality samples.

Data Generation Pipeline

The dataset was constructed using a multi-step process to ensure high quality and diversity:

  1. Data Gathering: Approximately 50,000 multilingual documents were scraped from the public internet using topic-based searches. To ensure diversity, pages were classified as text-only, visual-only, or mixed using a document layout analysis model.
  2. Synthetic Query Generation: Queries were generated using Gemini 1.5 Pro and Qwen2-VL-72B. The models were tasked with generating both specific and general questions to improve the strength of the specific questions used for training.
  3. Cleaning and Filtering: Queries underwent a cleaning process to ensure correct language, proper formatting, and the removal of grounding phrases (e.g., "according to Figure 1").
  4. Hard-Negative Mining: To improve retrieval precision, hard negatives were mined using the voyage-3 embedding model. A query was considered "good" if its associated broad question appeared in the top 100 results of an index.

Performance and Evaluation

vdr-2b-multi-v1 was evaluated on the ViDoRe benchmark and a custom multilingual test set. The model demonstrates significant performance gains over the base model (dse-qwen2-2b-mrl-v1), particularly in non-English visual-only and mixed page types.

Benchmark Results (NDCG@5)

The multilingual model outperforms the base model across all tested languages by an average of 2.2%. Notable improvements include:

  • German (Visual-only): +6.33% improvement over the base model.
  • Italian (Mix): +2% improvement.
  • Spanish (Visual-only): +1.4% improvement.

Cross-Lingual Performance

The model demonstrates effective cross-lingual retrieval capabilities. In tests where German evaluation sets were queried using Italian translations, the model showed an average improvement of 2.3% over the base model across all document types.

Implementation and Integration

The models are compatible with standard machine learning workflows via SentenceTransformers and LlamaIndex.

For LlamaIndex users, integration is handled through the llama-index-embeddings-huggingface package. For SentenceTransformers, the model supports torch_dtype=torch.bfloat16 and flash_attention_2 for optimized performance on CUDA devices.

Sources