SmolLM Release: High-Performance Small Language Models
Hugging Face has announced SmolLM, a family of state-of-the-art small language models (SLMs) available in three sizes: 135M, 360M, and 1.7B parameters. These models are designed for local deployment on devices such as smartphones and laptops, reducing inference costs and improving user privacy by achieving high performance through high-quality data curation.
Data Curation and the SmolLM-Corpus
The performance of SmolLM is driven by the SmolLM-Corpus, a specialized training set consisting of three primary components:
- Cosmopedia v2: A collection of 39 million synthetic documents (28B tokens) including textbooks, stories, and articles. It was generated by Mixtral-8x7B-Instruct-v0.1 using a refined prompting strategy based on 34,000 topics from the BISAC book classification system.
- FineWeb-Edu (deduplicated): 220B tokens of educational web samples filtered from the FineWeb dataset using an educational quality classifier.
- Python-Edu: 4B tokens of educational Python samples derived from The Stack, filtered via a Llama3-trained educational code classifier.
Improving Synthetic Data with Cosmopedia v2
To enhance Cosmopedia v2 over its predecessor, Hugging Face focused on prompt optimization. Instead of unsupervised clustering, they utilized the BISAC book classification to define 34,000 topics and implemented a search tool to retrieve the most relevant web pages as seed samples.
Research into generation styles revealed that textbooks targeted at middle school students provided the best performance across most benchmarks (excluding MMLU), while stories improved common sense reasoning. Consequently, the final mix for v2 consists of 40% middle school content, 30% college content, and 30% mixed styles.
Model Architecture and Training
SmolLM models were trained using a trapezoidal learning rate scheduler with a 20% cooldown phase. The training volume varied by model size:
- 135M and 360M models: Trained on 600B tokens.
- 1.7B model: Trained on 1T tokens.
Technical Specifications
| Model Size | Architecture Notes | Context Length | Vocab Size |
|---|---|---|---|
| 135M | Grouped-Query Attention (GQA), Depth-prioritized | 2048 | 49,152 |
| 360M | Grouped-Query Attention (GQA), Depth-prioritized | 2048 | 49,152 |
| 1.7B | Traditional Architecture | 2048 | 49,152 |
All models utilize embedding tying. The 135M and 360M models follow a design similar to MobileLLM to optimize for local device constraints.
Performance Evaluation
SmolLM models outperform several competing small language models across common sense reasoning and world knowledge benchmarks:
- SmolLM-135M outperforms MobileLM-125M despite being trained on fewer tokens (600B vs 1T).
- SmolLM-360M outperforms all models under 500M parameters, including Qwen2-500M and MobileLM-350M.
- SmolLM-1.7B outperforms other models under 2B parameters, including Microsoft's Phi-1.5, Qwen2-1.5B, and MobileLM-1.5B.
In coding tasks, SmolLM-1.7B achieved a 24 pass@1 score on HumanEval.
Instruction Tuning and Alignment
Instruction-tuned versions were created using the permissive subset of the WebInstructSub dataset and StarCoder2-Self-OSS-Instruct. Alignment was further refined using Direct Preference Optimization (DPO) with HelpSteer (for 135M and 1.7B) and argilla/dpo-mix-7k (for 360M).
Local Deployment and Hardware Requirements
SmolLM is optimized for low-memory environments. The models are compatible with various hardware, including iPhones (which typically have 6GB to 8GB of DRAM). Hugging Face has released transformers checkpoints, ONNX checkpoints, and plans to provide GGUF versions for llama.cpp compatibility.