Comparing RoBERTa, Llama 2, and Mistral for Disaster Tweet Classification with LoRA
This study compares the performance of RoBERTa, Llama 2, and Mistral 7B on a disaster tweet classification task using Low-Rank Adaptation (LoRA). The results demonstrate that the smaller RoBERTa model outperforms the larger 7B-parameter models in both accuracy (F1 score) and efficiency, suggesting that massive LLMs may not be necessary for simple short-sequence binary classification tasks.
Comparative Performance Results
RoBERTa achieved the highest F1 score and the lowest resource consumption among the three tested models. Llama 2 performed better than Mistral 7B, but both were significantly slower and more memory-intensive than RoBERTa.
| Model | F1 Score | Training Time | Memory Consumption | Trainable Parameters |
|---|---|---|---|---|
| RoBERTa | 0.8077 | 538 seconds | GPU1: 9.1 Gb / GPU2: 8.3 Gb | 0.64% |
| Llama 2 | 0.7638 | 2052 seconds | GPU1: 35 Gb / GPU2: 33.9 Gb | 0.12% |
| Mistral 7B | 0.7364 | 2030 seconds | GPU1: 29.6 Gb / GPU2: 29.5 Gb | 0.024% |
Technical Implementation and Methodology
Model Architectures
Three distinct architectures were evaluated to determine the impact of model size and type on classification performance:
- RoBERTa (Large): An encoder-only transformer model with 355M parameters, used as the baseline.
- Llama 2 (7B): An auto-regressive decoder-based model featuring SwiGLU activation and rotary positional embeddings.
- Mistral 7B (v0.1): A decoder-based model utilizing Sliding Window Attention and Grouped-query Attention to optimize inference and handle longer sequences.
LoRA Fine-Tuning
Low-Rank Adaptation (LoRA) was used to reduce the number of trainable parameters by learning low-rank update matrices while keeping pre-trained weights frozen. This Parameter-Efficient Fine-Tuning (PEFT) approach was applied to both encoder and decoder architectures.
For the decoder models (Llama 2 and Mistral 7B), LoRA was specifically targeted at the q_proj and v_proj modules. The resulting percentage of trainable parameters was extremely low: 0.024% for Mistral 7B and 0.12% for Llama 2.
Dataset and Preprocessing
The models were trained on the mehdiiraqui/twitter_disaster dataset. To ensure a fair comparison, a maximum sequence length (MAX_LEN) of 512 was set for all models, constrained by RoBERTa's limit.
Because the dataset exhibited an imbalanced distribution of positive and negative classes, a custom WeightedCELossTrainer was implemented. This trainer overrides the compute_loss method to apply weighted cross-entropy loss, using calculated weights (Positive: 1.1637, Negative: 0.8767) to prevent model bias toward the majority class.
Hardware and Training Configuration
Training was conducted on a single node with one A6000 GPU (48GB memory).
Due to the size of Mistral 7B and Llama 2, half-precision training (fp16=True) was required to fit the models into GPU memory, whereas RoBERTa was trained without half-precision. Both 7B models utilized gradient checkpointing to further manage memory usage.
Key Takeaways
- Model Size vs. Task Complexity: For simple prediction tasks involving short sequences, smaller base models like RoBERTa remain highly competitive and often superior to larger LLMs.
- Efficiency Gains: RoBERTa was approximately 3.8x faster to train than the 7B models while using significantly less memory.
- LoRA Versatility: The study confirms that LoRA is effective for sequence classification across both encoder-only and decoder-only transformer architectures.