Synthetic Data with Open-Source LLMs Cuts Cost, Latency, and Carbon for Custom Models

TL;DR

Hugging Face demonstrates that synthetic data generated by the open‑source Mixtral‑8x7B‑Instruct model can be used to fine‑tune a tiny RoBERTa‑base classifier that matches GPT‑4’s 94 % accuracy on investor‑sentiment analysis while costing only $2.7, emitting 0.12 kg CO₂, and responding in 0.13 s for 1 M sentences.

1. The data gap problem

Companies often lack task‑specific labeled data.

  • Public datasets on the Hugging Face Hub cover generic sentiment (Twitter, poems, etc.) but not niche domains such as brand‑specific financial sentiment.
  • Without suitable data, teams either (a) build their own annotation pipeline (expensive, time‑consuming) or (b) rely on closed‑source LLM APIs (easy but costly, opaque, and create data‑privacy dependencies).

2. Synthetic data as a solution

LLMs now reach human‑level annotation quality.

  • Recent research (Zheng et al., 2023; Gilardi et al., 2023; He et al., 2023) shows top‑tier LLMs outperform crowd workers and match expert annotators.
  • The bottleneck shifts from hiring annotators to providing clear prompts; compute becomes the only limiting factor.
  • Open‑source models with permissive licenses (e.g., Mixtral‑8x7B‑Instruct‑v0.1, Apache 2.0) allow commercial use of generated synthetic data, eliminating legal uncertainty that plagues OpenAI‑generated outputs.

3. End‑to‑end case study: Financial‑sentiment monitoring

3.1 Prompting an LLM to annotate data

  • The financial_phrasebank dataset (2 264 sentences, three‑class investor sentiment) serves as a testbed.
  • A concise instruction prompt asks the LLM to label each sentence as positive, negative, or neutral without explanations.
  • The prompt is wrapped in Mixtral’s chat template via tokenizer.apply_chat_template and sent to the free Hugging Face Inference API (or a dedicated endpoint for larger workloads).
  • Simple post‑processing (clean_output) maps raw LLM strings to the three canonical labels.

3.2 Improving annotation quality with CoT & Self‑Consistency

  • Chain‑of‑Thought (CoT): the LLM first reasons step‑by‑step, then outputs a JSON object containing reason and label.
  • Self‑Consistency (SC): the same CoT prompt is run three times; the majority vote among the three labels is taken as the final annotation.
  • This combination raises accuracy from 91.6 % (plain prompt) to 94.0 % and F1‑macro from 0.916 to 0.94, matching GPT‑4 on this task.

3.3 Benchmarking open‑source vs. proprietary LLMs

Model Prompt type Accuracy F1‑macro
Mixtral‑8x7B‑Instruct (CoT+SC) CoT+SC 94 % 0.94
GPT‑3.5‑turbo‑0613 CoT+SC lower than Mixtral (exact numbers omitted)
GPT‑4‑0125‑preview CoT+SC comparable to Mixtral
  • The table (in the original blog) shows Mixtral outperforming GPT‑3.5 and achieving parity with GPT‑4 for this specific annotation task.

3.4 Validating synthetic annotations

  • Validation tools such as Argilla, LabelStudio, and CleanLab help spot noisy or ambiguous labels.
  • Human review remains essential; even expert gold standards contain disagreement (Krippendorff 2004; Hosking et al., 2024).

3.5 Fine‑tuning a specialized student model with AutoTrain

  • Synthetic labels are saved as a CSV (text, labels).
  • Hugging Face AutoTrain (no‑code UI) fine‑tunes a RoBERTa‑base (≈0.13 B parameters) on the 1 811‑sample training split.
  • Training takes ~15 minutes on an A10G GPU ($1.05/h) and costs < $1.
  • The resulting model reaches 94 % accuracy, identical to its teacher Mixtral and to GPT‑4, while being orders of magnitude smaller.

4. Comparative trade‑offs

Approach Performance Inference cost (1 M sentences) Latency Development effort Data control CO₂ impact
Manual data + custom model High (task‑specific) $2.7 (RoBERTa) 0.13 s High (data collection, QA, fine‑tuning) Full (on‑prem) 0.12 kg
LLM API only High (general) $3 061 (GPT‑4) > seconds Low (just prompts) None (data sent to provider) 0.735‑1.1 t
Synthetic data → fine‑tuned model (this work) High (parity) $2.7 0.13 s Medium (prompt design, AutoTrain) Full (synthetic data owned) 0.12 kg
  • The table (originally an image) illustrates that the synthetic‑data pipeline delivers GPT‑4‑level accuracy at a fraction of the cost, speed, and carbon footprint.

5. Broader implications

  • Cost efficiency: Inference cost drops from thousands of dollars to a few dollars for large‑scale workloads.
  • Speed: Specialized models process sentences in ~0.13 s versus multi‑second latencies of large LLM APIs.
  • Environmental impact: Specialized models emit ~0.12 kg CO₂ for 1 M sentences, compared to ~0.735‑1.1 t for GPT‑4.
  • Control & compliance: Synthetic data generated from Apache‑2.0 models can be freely used for commercial training, avoiding the legal gray area of OpenAI’s Business Terms.
  • Scalability: The same pipeline adapts to other classification tasks (customer intent, toxic content), token‑level tasks (NER, PII), or generative tasks (summarization, QA).

Conclusion

Hugging Face’s demonstration proves that open‑source LLMs can serve as high‑quality annotators, enabling the creation of synthetic datasets that power tiny, efficient student models. This three‑step workflow—prompt‑based annotation, validation, and AutoTrain fine‑tuning—delivers GPT‑4‑level accuracy while slashing compute costs, latency, and carbon emissions, empowering companies to build controllable, sustainable AI solutions without sacrificing performance.

All code, notebooks, and reproducibility scripts are available in the public GitHub repository linked in the original blog post.

Sources