SetFitABSA: Few-Shot Aspect Based Sentiment Analysis
Hugging Face and Intel Labs have introduced SetFitABSA, a framework designed for the few-shot training of domain-specific Aspect-Based Sentiment Analysis (ABSA) models. SetFitABSA enables the detection of sentiment toward specific aspects within a text using only a handful of labeled examples, often matching or exceeding the performance of significantly larger generative models such as Llama 2 and T5.
Advantages Over LLM-Based Methods
SetFitABSA provides two primary advantages over Large Language Model (LLM) methods that rely on in-context learning:
- Elimination of Prompts: Unlike LLMs, which require handcrafted prompts that can be brittle and sensitive to phrasing, SetFitABSA generates rich embeddings directly from a small number of labeled text examples.
- Faster Training and Labeling: The framework uses a simple training data format that does not require specialized tagging tools, making the data labeling process faster and easier.
Technical Architecture: The Three-Stage Process
SetFitABSA operates through a sequential three-step pipeline to identify aspects and their associated sentiments.
1. Aspect Candidate Extraction
The system assumes that aspects (typically product or service features) are primarily nouns or noun compounds. It uses spaCy to tokenize text and extract all nouns and noun compounds from the few-shot training set, which are then treated as "aspect candidates."
2. Aspect/Non-Aspect Classification
To distinguish actual aspects from general nouns, a binary classifier is trained using a SetFit model. To adapt the sentence-level SetFit framework for token-level classification, each aspect candidate is concatenated with the full sentence using the template aspect_candidate:training_sentence.
- True Labels: Aspects identified in the training set.
- False Labels: Non-overlapping candidate nouns that are not aspects.
3. Sentiment Polarity Classification
Once aspects are extracted, a second SetFit model is trained to associate a sentiment polarity (e.g., positive, negative, or neutral) with each aspect. This model is trained similarly to the extraction model, using the same aspect_candidate:training_sentence template, but it only includes confirmed aspects in the training set to focus exclusively on polarity classification.
Benchmarking and Performance
SetFitABSA was evaluated using the Laptop14 and Restaurant14 datasets from the SemEval 2014 Challenge, focusing on aspect term extraction (SB1) and the combined task of extraction and polarity prediction (SB1+SB2).
Model Size Efficiency
SetFitABSA is significantly smaller than the generative models it competes with. While Llama-2-chat contains 7B parameters, SetFitABSA (using MPNet) utilizes two 110M parameter models (totaling 220M parameters for the full task).
| Model | Size (params) |
|---|---|
| Llama-2-chat | 7B |
| T5-base | 220M |
| GPT2-base | 124M |
| GPT2-medium | 355M |
| SetFit (MPNet) | 2x 110M |
Performance Results
In low-data scenarios, SetFitABSA demonstrates a clear performance advantage over T5 and GPT2. When compared to Llama 2, SetFitABSA's performance is on par or better, despite Llama 2 being 64 times larger. Notably, the researchers observed that increasing the number of in-context training samples for Llama 2 did not consistently improve its performance.
Implementation and Training
SetFitABSA is integrated into the setfit library. Training requires the setfit[absa] package and the en_core_web_lg spaCy model.
Data Requirements
The training dataset must be a Dataset object containing four columns:
- text: The full sentence.
- span: The specific aspect term (e.g., "food").
- label: The polarity label (e.g., "positive").
- ordinal: The index of the occurrence if the aspect appears multiple times (usually 0).
Training Workflow
Training involves initializing an AbsaModel from a Sentence Transformer (such as sentence-transformers/paraphrase-mpnet-base-v2) and using the AbsaTrainer to execute the training. Because the process involves two sequential models, GPU acceleration is recommended; a full model can be trained in approximately 10 minutes on a Google Colab T4 GPU.