Hugging Face Sentence Transformers Training Guide (historical reference)

TL;DR

Hugging Face published a historical guide that walks through building, training, and fine‑tuning Sentence Transformers models, covering architecture, dataset preparation, loss selection, and model publishing, but notes that the SentenceTransformer.fit API described is obsolete and points readers to the newer SentenceTransformerTrainer‑based guides.

Overview of the Guide

The tutorial is retained for reference only; it explains how to create a Sentence Transformers model from scratch or fine‑tune an existing one, how to format training data, which loss functions match each format, and how to push the resulting model to the Hugging Face Hub.

Note: The guide uses the pre‑v3.0 SentenceTransformer.fit API, which has been superseded by SentenceTransformerTrainer. Current training procedures are documented in the following up‑to‑date posts:

  • Embedding models – Training and Finetuning Embedding Models with Sentence Transformers
  • Reranker models – Training and Finetuning Reranker Models with Sentence Transformers
  • Sparse embedding models – Training and Finetuning Sparse Embedding Models with Sentence Transformers
  • Multimodal models – Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

How Sentence Transformers Models Work

Sentence Transformers map variable‑length text (or images) to a fixed‑size embedding that captures semantic meaning.

  1. Transformer layer – Input text is processed by a pre‑trained Transformer (e.g., distilroberta-base). The model outputs contextualized token embeddings.
  2. Pooling layer – Token embeddings are aggregated (e.g., mean pooling) into a single sentence‑level vector.
from sentence_transformers import SentenceTransformer, models

# Layer 1: pre‑trained transformer
word_embedding_model = models.Transformer('distilroberta-base')

# Layer 2: pooling to a fixed‑size vector
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension())

# Assemble the modules
model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

The model is a sequential list of modules; additional layers (dense, convolutional, etc.) can be inserted if needed.

Why not use a vanilla Transformer for sentence embeddings?

  • Inference for semantic search with a raw BERT model on 10,000 sentences requires 50 million operations (65 h), whereas a Sentence Transformer reduces this to ~5 s.
  • Directly averaging BERT token embeddings yields poorer sentence representations than classic GloVe embeddings.

Preparing Your Dataset

Training requires a signal that two sentences are similar or dissimilar. The guide identifies four common dataset structures:

Case Format Typical Sources Recommended Loss
1 (sentence_a, sentence_b, similarity_label) – label may be integer or float Natural Language Inference (NLI) datasets ContrastiveLoss, SoftmaxLoss, CosineSimilarityLoss
2 (sentence_a, sentence_b) – positive pair, no explicit label Paraphrase, summary, duplicate‑question pairs MultipleNegativesRankingLoss, MegaBatchMarginLoss
3 (sentence, class_id) – integer class label Topic classification datasets (e.g., TREC) Triplet‑based losses that use class IDs (BatchHardTripletLoss, etc.)
4 (anchor, positive, negative) – explicit triplet, no class IDs Pre‑constructed triplet datasets (e.g., Quora Triplets) TripletLoss

The tutorial demonstrates case 4 using the embedding-data/QQP_triplets dataset. It shows how to load the dataset with datasets.load_dataset, inspect its structure, and convert each example to a sentence_transformers.InputExample:

from datasets import load_dataset
from sentence_transformers import InputExample

dataset = load_dataset('embedding-data/QQP_triplets')
train_examples = []
train_data = dataset['train']['set']
for i in range(dataset['train'].num_rows // 2):  # use half the data for speed
    ex = train_data[i]
    train_examples.append(
        InputExample(texts=[ex['query'], ex['pos'][0], ex['neg'][0]])
    )

The examples are then wrapped in a torch.utils.data.DataLoader for batching:

from torch.utils.data import DataLoader
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)

Selecting a Loss Function

The loss must align with the dataset format:

  • Case 1 – use ContrastiveLoss (integer labels) or CosineSimilarityLoss (float labels).
  • Case 2 – use MultipleNegativesRankingLoss (most common) or MegaBatchMarginLoss.
  • Case 3 – use triplet‑based losses that rely on class IDs, such as BatchHardTripletLoss.
  • Case 4 – use TripletLoss, which does not require class labels.

The code to instantiate a loss is minimal:

from sentence_transformers import losses
train_loss = losses.TripletLoss(model=model)

Training / Fine‑Tuning the Model

With a DataLoader and loss ready, training proceeds with a single fit call:

model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=10)

If fine‑tuning an existing model (e.g., sentence-transformers/all-MiniLM-L6-v2), load it via SentenceTransformer(model_id) and call fit directly.

Publishing the Model to the Hub

After training, push the model to the Hugging Face Hub:

from huggingface_hub import notebook_login
notebook_login()  # or `huggingface-cli login` in a terminal

model.save_to_hub(
    "distilroberta-base-sentence-transformer",
    organization="<your‑username-or‑org>",
    train_datasets=["embedding-data/QQP_triplets"]
)

save_to_hub automatically creates a model card, inference widget, and example snippets.

Limitations of Sentence Transformers

Sentence Transformers excel at semantic search and similarity tasks but are unsuitable for pure classification problems. For classification, the standard 🤗 Transformers library (e.g., sequence‑classification pipelines) should be used instead.

Additional Resources

  • Getting Started With Embeddings – introductory guide to embeddings.
  • Understanding Semantic Search – deep dive into semantic retrieval.
  • Your First Sentence Transformers Model – step‑by‑step beginner tutorial.
  • Playlist Generator – example application of Sentence Transformers.
  • Hugging Face + Sentence Transformers documentation – comprehensive API reference.

This guide is retained for historical reference only; consult the newer training guides that use SentenceTransformerTrainer for production‑ready workflows.

Sources