SmolLM3 Release: Multilingual, Long-Context 3B Reasoner

Hugging Face has introduced SmolLM3, a 3B parameter open language model designed for efficiency and high performance. SmolLM3 outperforms Llama-3.2-3B and Qwen2.5-3B and remains competitive with larger 4B models, providing a dual-mode reasoning capability that allows users to toggle between explicit thinking and direct answers.

Architecture and Pretraining

SmolLM3 utilizes a transformer decoder architecture with tied embeddings, incorporating several modifications to optimize for efficiency and long-context performance:

  • Grouped Query Attention (GQA): Replaces multi-head attention with 4 groups to reduce KV cache size during inference without sacrificing performance.
  • NoPE: Selectively removes rotary position embeddings from every 4th layer to improve long-context performance while maintaining short-context capabilities.
  • Intra-Document Masking: Prevents tokens from different documents in a single training sequence from attending to each other, ensuring more stable long-context training.
  • Training Stability: Weight decay is removed from embedding layers to stabilize training dynamics.

Three-Stage Pretraining

The model was trained on 11.2T tokens using a three-stage strategy to progressively boost domain performance:

  1. Stable Phase (0T to 8T tokens): Focuses on general capabilities with a mixture of 85% Web (including 12% multilingual), 12% Code, and 3% Math.
  2. Stable Phase (8T to 10T tokens): Increases high-quality math and code data, shifting to 75% Web, 15% Code, and 10% Math.
  3. Decay Phase (10T to 11.1T tokens): Further upsamples math and code, ending with 63% Web, 24% Code, and 13% Math, while introducing instruction and reasoning datasets like OpenMathReasoning.

Mid-Training: Context and Reasoning

Following the main pretraining, SmolLM3 underwent "mid-training" to enhance specific capabilities before final supervised fine-tuning.

Long Context Extension

To extend the context window, the model was trained on an additional 100B tokens in two stages: transitioning from 4k to 32k (RoPE theta 1.5M), and then from 32k to 64k (RoPE theta 5M). By using YARN for extrapolation during inference, SmolLM3 can handle up to 128k context.

Reasoning Adaptation

SmolLM3 was trained on 35B tokens of general reasoning data from OpenThoughts3-1.2M and a subset of NVIDIA's Llama-Nemotron-Post-Training-Dataset-v1.1. This stage focused on teaching the model to reason generally across domains rather than targeting specific subjects like math or code.

Post-Training and Dual-Mode Reasoning

SmolLM3 is a dual-instruction model that supports both reasoning (/think) and non-reasoning (/no_think) modes.

Supervised Fine-Tuning (SFT)

The SFT dataset consists of 1.8B tokens (1B non-reasoning, 0.8B reasoning). To fill gaps in reasoning traces for specific domains, Hugging Face generated synthetic data by prompting Qwen3-32B in reasoning mode with non-reasoning prompts. This improved performance in multi-turn conversations and multilinguality.

Alignment with Anchored Preference Optimization (APO)

Alignment was performed using Anchored Preference Optimization (APO), a more stable variant of Direct Preference Optimization (DPO). The process used the Tulu3 preference dataset for non-reasoning mode and synthetic pairs (chosen from Qwen3-32B, rejected from Qwen3-0.6B) for reasoning mode.

Model Merging for Recovery

Because the reasoning mid-training and APO stages caused a performance drop on long-context benchmarks (RULER), Hugging Face used MergeKit to perform a linear merge. They combined an APO model "soup" with a mid-training checkpoint (weights 0.9 and 0.1, respectively), recovering the base model's RULER scores up to 128k tokens.

Performance Evaluation

Base Model

SmolLM3 consistently outperforms other 3B models across 12 benchmarks covering knowledge, reasoning, math, and coding (including HellaSwag, ARC, and GSM8K). It is competitive with 4B models such as Qwen3-4B and Gemma3-4B and shows strong multilingual performance across five major European languages.

Instruct and Reasoning Model

  • Non-Reasoning Mode: SmolLM3 outperforms Llama-3.2-3B Instruct and Qwen2.5-3B Instruct, positioning it as a high-efficiency alternative to larger reasoning models.
  • Reasoning Mode: With extended thinking enabled, the model shows significant gains in complex tasks. For example, AIME 2025 scores increased from 9.3% (non-reasoning) to 36.7% (reasoning), and LiveCodeBench scores rose from 15.2% to 30.0%.

Local Implementation and Usage

SmolLM3 requires transformers v4.53.0 or later. Users can control the reasoning mode via the system prompt:

  • Extended Thinking: Include the /think flag in the system prompt.
  • Direct Answer: Include the /no_think flag in the system prompt.

The model also supports tool calling via xml_tools (standard) and python_tools (Python function snippets) arguments in the chat template.

Sources