Hugging Face CFM Case Study: Fine-tuning Small Models with LLM Insights
Capital Fund Management (CFM) successfully optimized Named Entity Recognition (NER) for financial data by using large language models (LLMs) to label training data for smaller, more efficient models. This approach improved accuracy by up to 6.4% and reduced operational costs by up to 80x compared to relying solely on large LLMs for inference.
LLM-Assisted Labeling Workflow
CFM utilized a pipeline combining LLM automation and human verification to create high-quality datasets for fine-tuning. The process involved the following steps:
Synthetic Label Generation
CFM deployed Llama 3.1-70b-Instruct using Hugging Face Inference Endpoints. To ensure structured and accurate output, the team used a specific system prompt defining the model as a financial expert and implemented a Pydantic schema to force the model to return JSON containing both the raw extracted company names and their normalized versions.
Processing 900,000 samples from the Financial News and Stock Price Integration Dataset (FNSPID) took approximately 8 hours and cost roughly $70.
Human-in-the-Loop Refinement
To ensure the ground truth was reliable, CFM used Argilla, an open-source data annotation platform. The team sampled 2,714 headlines based on company clusters created via fuzzy matching.
By using the pre-computed Llama labels as a starting point, human annotators reduced the time spent per sample from 30 seconds to 5-10 seconds. The entire annotation of 2,714 samples was completed in approximately 8 hours.
Zero-Shot Performance Comparison
Before fine-tuning, CFM evaluated several models in a zero-shot setting to establish a performance baseline on the curated dataset:
| Model | F1-Score (Zero-Shot) |
|---|---|
| Llama 3.1-70b | 95% |
| Llama 3.1-8b | 88% |
| GLiNER | 87% |
| SpanMarker | 47% |
While Llama 3.1-70b provided the highest accuracy, its computational requirements are significantly higher than those of the compact models.
Impact of Fine-Tuning on Compact Models
Fine-tuning smaller models on the LLM-assisted human-verified dataset resulted in performance gains that rivaled the largest LLMs while drastically reducing costs.
Performance Gains
Fine-tuning transformed the performance of the smaller models:
- GLiNER-medium-news improved from 87.0% to 93.4% F1-score.
- SpanMarker improved from 47.0% to 90.1% F1-score.
Cost and Efficiency
The cost difference between the fine-tuned small models and the large LLMs is substantial:
- Llama 3.1-70b costs at least $8 per hour for inference.
- Compact models (GPU) cost approximately $0.50 per hour (16x cheaper).
- Compact models (CPU) cost approximately $0.10 per hour (80x cheaper).
Weak Supervision vs. Human-Annotated Data
CFM compared the results of fine-tuning GLiNER on human-annotated data versus synthetic data generated by Llama 3.1-70b (weak supervision).
- Accuracy: Models trained on human-annotated data consistently achieved higher F1-scores as the dataset size increased.
- Trade-off: For 1,000 samples, manual annotation took 3 hours and yielded an F1 score of 0.915, while Llama 3.1-70b inference took 2 minutes and yielded an F1 score of 0.895.
The study concludes that while LLM-assisted labeling is highly efficient for scaling, human verification remains critical for achieving maximum accuracy.