Sentence Transformers v5.4: Multimodal Embedding and Reranker Models

Hugging Face announced that Sentence Transformers v5.4 adds multimodal capabilities, enabling the encoding and comparison of text, images, audio, and video using a unified API for embedding and reranking tasks. This extends the library to support cross-modal search, visual document retrieval, and multimodal RAG pipelines without requiring users to learn a new API.

Installation

To use multimodal models in Sentence Transformers v5.4, install the extra dependencies for the desired modalities (image, audio, video) via pip, noting that VLM-based models require a GPU with sufficient VRAM for practical use.

Install image support with pip install -U "sentence-transformers[image]", audio support with pip install -U "sentence-transformers[audio]", and video support with pip install -U "sentence-transformers[video]". Mix and match as needed, for example pip install -U "sentence-transformers[image,video,train]". VLM-based models like Qwen3-VL-2B require a GPU with at least ~8 GB of VRAM; for 8B variants, expect ~20 GB. Without a local GPU, consider cloud GPU services or Google Colab, as CPU inference will be extremely slow for these models.

Multimodal Embedding Models

Multimodal embedding models map inputs from different modalities into a shared embedding space, enabling cross-modal similarity comparisons using the same API as text-only models.

Loading a Model

Loading a multimodal embedding model works exactly like loading a text-only model using SentenceTransformer("model_name\)), with the model automatically detecting supported modalities. Some models may temporarily require a revision argument pending integration pull requests, after which it will no longer be needed.

Encoding Images

With a multimodal model loaded, model.encode() accepts images alongside text, provided as URLs, local file paths, or PIL Image objects. For example, encoding two images returns embeddings of shape (2, 2048) for the Qwen/Qwen3-VL-Embedding-2B model.

Cross-Modal Similarity

Text and image embeddings can be compared directly since they reside in the same space, though absolute similarity scores are typically lower than within-modal comparisons due to the modality gap. Relative ordering is preserved, so retrieval remains effective. For instance, a text query matching an image yields a higher similarity score than mismatched pairs.

Encoding Queries and Documents

For retrieval tasks, use encode_query() and encode_document() to automatically apply model-specific prompts for queries and documents. These methods are thin wrappers around encode() that handle prompt selection, accepting the same input types and parameters. If no specialized prompts exist, they behave identically to encode().

Multimodal Reranker Models

Multimodal reranker models score the relevance of mixed-modality pairs, often outperforming embedding models in quality but at higher computational cost due to pairwise processing.

Ranking Mixed-Modality Documents

The rank() method scores and ranks documents against a query, supporting mixed modalities such as images, text, and combined text-image documents. For example, a reranker correctly identifies the most relevant image document followed by a combined text-image document, with text-only and irrelevant image documents scoring lower. Modality support can be checked via the modalities property and supports() method.

Predicting Pair Scores

Use predict() to obtain raw relevance scores for specific input pairs, such as text-image combinations, returning numerical scores indicating relevance.

Retrieve and Rerank

A common pipeline uses an embedding model for fast initial retrieval over a large corpus, then refines the top results with a reranker for improved accuracy. Pre-compute corpus embeddings once for efficient retrieval, then apply the reranker only to the top-k candidates from the initial similarity search.

Input Formats and Configuration

Multimodal models accept diverse input formats per modality, with configuration options for preprocessing and model loading.

Supported Input Types

Text inputs are strings. Images accept PIL objects, file paths, URLs, numpy arrays, or torch tensors. Audio accepts file paths, URLs, numpy/torch arrays, dicts with array and sampling_rate, or torchcodec.AudioDecoder instances. Video accepts file paths, URLs, numpy/torch arrays, dicts with array and video_metadata, or torchcodec.VideoDecoder instances. Multimodal inputs are dicts mapping modality names (text, image, audio, video) to values. Message inputs are lists of message dicts with role and content keys, supporting structured (typed or flat formats.

Checking Modality Support

Use the modalities property to list supported modalities and the supports() method to check for a specific modality or modality pair. For example, `model.supports("image)) returns True for a vision-capable model.

Processor and Model kwargs

Control image resolution and model precision via processor_kwargs (passed to AutoProcessor) and model_kwargs (passed to the appropriate AutoModel class). For instance, set min_pixels and max_pixels in processor_kwargs to adjust image processing, and specify attn_implementation or torch_dtype in model_kwargs. In v5.4, tokenizer_kwargs was renamed to processor_kwargs (the old name remains accepted but deprecated).

Supported Models

Sentence Transformers v5.4 supports a range of multimodal embedding and reranker models, along with new text-only rerankers and continued CLIP model support.

Supported Multimodal Embedding Models

Supported embedding models include Qwen/Qwen3-VL-Embedding-2B (2B parameters, text/image/video), Qwen/Qwen3-VL-Embedding-8B (8B), nvidia/llama-nemotron-embed-vl-1b-v2 (1.7B, text/image), nvidia/omni-embed-nemotron-3b (4.7B, text/image), LCO-Embedding models covering text/image/audio/video (5B and 9B), BidirLM/BidirLM-Omni-2.5B-Embedding (2.5B, text/image/audio), and various BAAI/BGE-VL models (ranging from 0.1B to 8B parameters, text/image). Some models like eagerworks/eager-embed-v1 and nomic-ai/nomic-embed-multimodal-3b/7b require specific revisions (e.g., revision="refs/pr/2"), while Haon-Chen/e5-omni models for text/image/audio/video also require revisions.

Supported Multimodal Reranker Models

Supported reranker models include Qwen/Qwen3-VL-Reranker-2B (2B, text/image/video), Qwen/Qwen3-VL-Reranker-8B (8B), nvidia/llama-nemotron-rerank-vl-1b-v2 (2B, text/image), and jinaai/jina-reranker-m0 (2B, text/image).

Text-Only Reranker Models (also new in v5.4)

New text-only rerankers in v5.4 include Qwen/Qwen3-Reranker-0.6B (0.6B), Qwen/Qwen3-Reranker-4B (4B), Qwen/Qwen3-Reranker-8B (8B), mixedbread-ai/mxbai-rerank-base-v2 (0.5B), mixedbread-ai/mxbai-rerank-large-v2 (2B), and ContextualAI/ctxl-rerank-v2-instruct-multilingual models (1B, 3B, 6B revisions required for some).

CLIP Models

Older CLIP models remain supported, including sentence-transformers/clip-ViT-L-14 (75.4% ImageNet zero-shot top-1 accuracy), sentence-transformers/clip-ViT-B-16 (68.1%), sentence-transformers/clip-ViT-B-32 (63.3%), and sentence-transformers/clip-ViT-B-32-multilingual-v1 (multilingual text encoder, 50+ languages).

Additional Resources

For further learning, consult the official documentation on usage, pretrained models, and installation. Training guidance is available in the companion blogpost "Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers" and related guides for embedding, reranker, sparse encoder, Matryoshka, static embeddings, and quantization techniques. Models, datasets, and the v5.4 integrations collection are accessible via the Hugging Face Hub.