SetFit: Efficient Few-Shot Learning Without Prompts
Hugging Face, in collaboration with Intel Labs and the UKP Lab, has released SetFit, a framework for the efficient few-shot fine-tuning of Sentence Transformers. SetFit allows developers to achieve high classification accuracy with very few labeled examples—sometimes as few as 8 per class—without requiring the handcrafted prompts or verbalisers typically associated with few-shot learning.
Contrastive Learning Architecture
SetFit employs a two-stage training process to generate high-quality embeddings from limited data:
- Contrastive Fine-Tuning: The framework first fine-tunes a Sentence Transformer model using a small set of labeled examples. It creates positive and negative pairs (or triplets) through in-class and out-class selection, training the model to generate dense vectors that cluster by class.
- Classification Head Training: A classification head is then trained on the embeddings generated by the fine-tuned Sentence Transformer using the respective class labels.
At inference time, an unseen example is passed through the fine-tuned Sentence Transformer to generate an embedding, which the classification head then uses to predict the class label.
Performance and Benchmarking
SetFit demonstrates high sample efficiency and robustness to noise, often performing on par with or better than state-of-the-art few-shot methods despite using significantly smaller models.
RAFT Benchmark Results
On the RAFT few-shot classification benchmark, SetFit (using the all-roberta-large-v1 model with 355 million parameters) outperformed PET and GPT-3. It achieved an accuracy of 71.3%, placing it just below the 11 billion parameter T-Few model (75.8%) and the human baseline (73.5%). Notably, SetFit outperformed the human baseline on 7 of the 11 RAFT tasks.
| Rank | Method | Accuracy | Model Size |
|---|---|---|---|
| 2 | T-Few | 75.8 | 11B |
| 4 | Human Baseline | 73.5 | N/A |
| 6 | SetFit (Roberta Large) | 71.3 | 355M |
| 9 | PET | 69.6 | 235M |
| 11 | SetFit (MP-Net) | 66.9 | 110M |
| 12 | GPT-3 | 62.7 | 175 B |
Comparison with Other Methods
With only 8 examples per class, SetFit typically outperforms PERFECT, ADAPET, and vanilla fine-tuned transformers. It also achieves results comparable to T-Few 3B while remaining 27 times smaller and prompt-free.
Training and Inference Efficiency
SetFit is significantly faster and cheaper to train and run than large-scale few-shot models.
- Training Speed: Training SetFit on an NVIDIA V100 with 8 labeled examples takes approximately 30 seconds at a cost of $0.025. In contrast, training T-Few 3B on an NVIDIA A100 takes 11 minutes at a cost of approximately $0.70—a 28x increase in cost and time.
- Hardware Accessibility: SetFit can be trained on a single GPU (such as those in Google Colab) or even on a CPU within a few minutes.
- Inference: SetFit provides substantial speed-ups during inference, and model distillation can further increase these speed-ups by up to 123x.
Multilingual Support and Implementation
SetFit is compatible with any Sentence Transformer available on the Hugging Face Hub. By using a multilingual Sentence Transformer as the base model, SetFit can be used for text classification in multiple languages, including German, Japanese, Mandarin, French, and Spanish, in both in-language and cross-linguistic settings.
To implement SetFit, Hugging Face provides a setfit library. The training workflow involves:
- Installing the library via
pip install setfit. - Loading a pretrained Sentence Transformer (e.g.,
paraphrase-mpnet-base-v2). - Using the
SetFitTrainerto handle contrastive learning and the training of the classification head.