Training and Finetuning Sparse Embedding Models with Sentence Transformers

Hugging Face has released a detailed technical guide on using the Sentence Transformers library to train and finetune sparse embedding models. These models are essential for hybrid search scenarios, offering a middle ground between traditional lexical methods like BM25 and dense embedding models by providing interpretable, high-dimensional sparse representations.

Understanding Sparse Embedding Models

Sparse embedding models convert text into high-dimensional vectors (e.g., 30,000+ dimensions) where most values are zero. Unlike dense models, which use low-dimensional vectors where most values are non-zero, sparse models map active dimensions to specific tokens in the model's vocabulary.

Key Capabilities

  • Query and Document Expansion: Neural sparse models automatically expand text with semantically related terms (e.g., expanding "weather" to include "sunny" or "beautiful"), allowing them to overcome vocabulary mismatch problems.
  • Interpretability: Because each dimension corresponds to a token, users can decode embeddings to see exactly which words contribute to a similarity score.
  • Hybrid Search Potential: Sparse models are highly effective when combined with dense models to capture both exact lexical matches and semantic meaning.

Architecture Options for Sparse Encoders

Depending on the specific use case, developers can choose from several distinct architectures within the Sentence Transformers framework:

SPLADE

SPLADE models utilize a Masked Language Modeling (MLM) transformer followed by a SpladePooling module. This architecture is the default when providing a fill-mask model to the SparseEncoder class.

Inference-free SPLADE

This architecture uses a Router module to handle queries and documents differently. It employs a lightweight SparseStaticEmbedding for queries to ensure near-instantaneous inference, while documents are processed using a full MLM transformer and SpladePooling. This is ideal for applications where query latency is critical.

Contrastive Sparse Representation (CSR)

CSR models apply a SparseAutoEncoder module on top of a dense Sentence Transformer model. Unlike SPLADE, CSR embeddings do not have the same size as the base model's vocabulary, meaning they cannot be directly interpreted by decoding tokens, but they are highly effective for high-dimensional dense encoders.

The Finetuning Workflow

Finetuning allows sparse models to learn domain-specific terminology (e.g., recognizing that "cephalalgia" is a synonym for "headache"). The training process involves several core components:

Training Components

  1. Model: A pre-trained Sparse Encoder or a base model (like BERT or RoBERTa).
  2. Dataset: Data loaded via the datasets library from the Hugging Face Hub or local files (CSV, JSON, Parquet, etc.).
  3. Loss Function: Functions like SpladeLoss or CSRLoss that add sparsity regularization to a main loss function.
  4. Evaluator: Tools to assess performance using metrics like NDCG, MRR, or MAP. Available evaluators include SparseNanoBEIREvaluator and SparseTripletEvaluator.
  5. Trainer: The SparseEncoderTrainer class that integrates all components to execute the training loop.

Training Tips and Best Practices

  • Distillation: Stronger sparse models are often trained using distillation from a stronger teacher model (such as a Cross-Encoder) rather than training directly from text pairs.
  • Sparsity Monitoring: Models should be evaluated not just on retrieval accuracy, but also on the sparsity of their embeddings; low sparsity increases storage costs and retrieval latency.
  • Multi-Dataset Training: The SparseEncoderTrainer supports training on multiple datasets simultaneously using MultiDatasetBatchSamplers (e.g., ROUND_ROBIN or PROPORTIONAL).

Evaluation and Deployment

Performance in Hybrid Pipelines

Empirical results demonstrate that combining Sparse and Dense rankings (using methods like Reciprocal Rank Fusion) significantly outperforms either method alone. For example, in the NanoMSMARCO dataset, a hybrid approach can yield substantial increases in NDCG@10 over dense-only baselines.

Vector Database Integration

For production deployment, sparse embeddings can be indexed in vector databases such as Qdrant, OpenSearch, Elasticsearch, or Seismic. Qdrant, for instance, provides native support for sparse vectors, enabling efficient semantic search at scale.

Sources