Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

Hugging Face released a tutorial demonstrating how to train and finetune multimodal embedding and reranker models using the Sentence Transformers library, enabling users to adapt models like Qwen3-VL-Embedding-2B to specific tasks such as Visual Document Retrieval.

Why Finetune Multimodal Models?

Finetuning adapts general-purpose multimodal models to specific tasks, significantly improving retrieval performance. General-purpose models trained on diverse data rarely excel at any single task; for Visual Document Retrieval, finetuning on domain-specific data increased NDCG@10 from 0.888 to 0.947 in the author's experiment, outperforming larger models.

Key Components for Training Multimodal Sentence Transformers

Training multimodal Sentence Transformer models requires six components: model, dataset, loss function, training arguments, evaluator, and trainer. The pipeline uses the same SentenceTransformerTrainer as text-only training, with the key difference that datasets contain multiple modalities (text, image, audio, video) and the model's processor handles preprocessing automatically.

Model Preparation for Multimodal Training

To finetune an existing multimodal embedding model, load it with SentenceTransformer and optionally specify model_kwargs (for precision and attention implementation) and processor_kwargs (for image resolution bounds). Alternatively, start from a Vision-Language Model checkpoint; Sentence Transformers will attempt to infer modalities and set up the forward method and pooling. The Transformer module inspects the processor to determine available modalities, and Pooling is added automatically if needed. Verify supported modalities using model.modalities and model.supports("modality)).

Dataset Requirements for Multimodal Training

Dataset format must match the loss function's input requirements. Columns other than "label" or "score" are inputs, and their order must match the number of valid inputs for the loss function. Inputs can contain text (strings), image (PIL images, file paths, URLs, or numpy/torch arrays), audio, video, or multimodal dicts mapping modality names to values. The data collator automatically calls model.preprocess() to handle modality-specific preprocessing, eliminating manual tokenization or image processing.

Loss Functions for Multimodal Retrieval

For retrieval tasks, use CachedMultipleNegativesRankingLoss, which accepts (query, positive) pairs with any number of hard negative columns. It pushes each query's similarity to its positive up and to every negative down, using hard negatives from the dataset and in-batch negatives from other samples in the same batch. The "cached" variant enables large effective batch sizes via gradient caching, controlled by the mini_batch_size parameter. To produce embeddings that work at multiple dimensionalities, wrap the base loss with MatryoshkaLoss, which trains the model so truncating embeddings to smaller dimensions retains good performance.

Training Arguments and Configuration

Configure training with SentenceTransformerTrainingArguments. Key settings for multimodal training include: bf16=True for better numerical stability with VLMs, batch_sampler=BatchSamplers.NO_DUPLICATES to ensure in-batch negatives are unique samples, and setting per_device_train_batch_size to a value like 64 (feasible due to gradient caching in CachedMultipleNegativesRankingLoss). Set eval_strategy, save_strategy, and logging_steps to fractions (e.g., 0.1) to evaluate, save, and log every 10% of an epoch.

Evaluation Setup for Multimodal Retrieval

Use InformationRetrievalEvaluator to compute metrics like NDCG@10, MAP, and Recall@k. Build evaluation data from the dataset: map queries and corpus using integer IDs, add hard negatives to the corpus with offset IDs to avoid collisions, and define relevant documents as the positive document at the same index as each query. The evaluator takes text queries, a corpus of images (including hard negatives), and a query-to-relevant-documents mapping. Use batch_size=1 during evaluation to prevent out-of-memory issues with large VLMs.

Training Process and Results

The training script integrates model, dataset, loss, arguments, and evaluator via SentenceTransformerTrainer. Differences from text-only training include: passing model_kwargs and processor_kwargs during model loading, using CachedMultipleNegativesRankingLoss with mini_batch_size=1 to manage memory, and using an evaluator with images in the corpus and text as queries. In the Visual Document Retrieval example, finetuning Qwen3-VL-Embedding-2B for one epoch achieved an NDCG@10 of 0.947 on the evaluation set, improving over the base model's 0.888 and outperforming all tested VDR models, including those up to 4x larger. Matryoshka training enabled strong performance at lower dimensions: the finetuned model retained over 92% of peak NDCG@10 even at 64 dimensions (32x smaller than full 2048).

Training Multimodal Reranker Models

Finetune multimodal Cross Encoder (reranker) models using CrossEncoderTrainer and Cross Encoder-specific loss functions. The process mirrors embedding model training but uses CrossEncoder instead of SentenceTransformer and appropriate loss functions like BinaryCrossEntropyLoss. Architectural choices include Any-to-Any with LogitScore (using the multimodal language model to generate a token and compute log-odds) or Feature Extraction with Pooling and Dense (extracting the last token's hidden state and projecting to a score). Training examples are available in the Sentence Transformers repository.

Additional Resources

The Sentence Transformers repository provides multimodal training examples: Visual Document Retrieval, Multimodal Reranker (Any-to-Any), and Multimodal Reranker (Feature Extraction). Documentation covers training overviews, loss functions, dataset handling, and API references. Companion blogposts discuss multimodal inference, text-only embedding and reranker training, sparse encoder training, Matryoshka embeddings, static embeddings, embedding quantization, and multilingual Visual Document Retrieval.