Introduction to Matryoshka Embedding Models
Matryoshka Embedding models enable the creation of embeddings that can be truncated to smaller dimensions without significant loss in performance. This approach allows practitioners to dynamically scale their embedding solutions based on storage costs, processing speed, and accuracy requirements.
How Matryoshka Embeddings Work
Matryoshka embeddings are based on Matryoshka Representation Learning (MRL), a technique that incentivizes a model to store the most critical information in the earliest dimensions of an embedding vector. Much like Russian nesting dolls, the model is trained to ensure that a smaller, truncated version of the embedding still retains enough information to be useful for downstream tasks.
Applications and Benefits
Variable-size embeddings provide two primary advantages for AI practitioners:
- Shortlisting and Reranking: Users can perform an initial, highly efficient "shortlist" search using small, truncated embeddings. Once a candidate set is identified, the full-dimensional embeddings can be used to rerank the results for higher precision.
- Resource Optimization: Matryoshka models allow for a customizable trade-off between storage costs, processing speed, and performance, depending on the specific constraints of the deployment environment.
Training Matryoshka Embedding Models
Theoretical Framework
In standard embedding training, a loss function is applied to the full-size embeddings. In contrast, Matryoshka training involves calculating loss values for the full-size embeddings as well as for embeddings at various truncated dimensionalities (e.g., 768, 512, 256, 128, and 64). These losses are summed to create a final loss value, which the optimizer uses to adjust model weights, effectively "frontloading" the most important information.
Implementation in Sentence Transformers
Sentence Transformers has implemented support for Matryoshka models via MatryoshkaLoss. This wrapper loss function applies a base loss (such as CoSENTLoss) to multiple truncated portions of the embeddings. Training with MatryoshkaLoss does not incur a notable overhead in training time.
Using Matryoshka Embedding Models
General Inference
Inference works similarly to standard embedding models. The only difference is that embeddings can be optionally truncated to a smaller dimensionality after they are produced. If the embeddings were originally normalized, they should be re-normalized after truncation to maintain consistency.
Implementation in Sentence Transformers
Using the SentenceTransformer class, users can specify the truncate_dim argument during model initialization. The encode function then automatically truncates the output embeddings to the specified size.
For specific models like nomic-ai/nomic-embed-text-v1.5, a custom architecture requires the application of F.layer_norm before the embedding is truncated.
Performance Results
Experimental results comparing a Matryoshka model (tomaarsen/mpnet-base-nli-matryoshka) against a standard model (tomaarsen/mpnet-base-nli) on the STSBenchmark test set demonstrate the following:
- Higher Accuracy: The Matryoshka model achieved a higher Spearman similarity across all tested dimensionalities.
- Performance Retention: The Matryoshka model's performance degrades much more slowly than that of a standard model. Even when truncated to 8.3% of its original size, the Matryoshka model preserved 98.37% of its maximum performance, compared to 96.46% for the standard model.
These results indicate that Matryoshka embeddings can significantly reduce storage and increase retrieval speed without a notable hit in performance.