Qwen3 Release Notes: Hybrid Thinking and Multilingual MoE Models

Qwen3 introduces a hybrid reasoning architecture that allows users to toggle between a deep "Thinking Mode" for complex problem-solving and a "Non-Thinking Mode" for rapid responses. The flagship model, Qwen3-235B-A22B, is designed to compete with top-tier models like DeepSeek-R1, o1, o3-mini, Grok-3, and Gemini-2.5-Pro in coding, mathematics, and general capabilities.

Model Lineup and Architecture

Qwen3 is available as both Mixture-of-Experts (MoE) and dense models under the Apache 2.0 license. The MoE models are designed for efficiency, with the base models achieving performance similar to Qwen2.5 dense models while using only 10% of the active parameters.

MoE Models

Model Total Parameters Activated Parameters Layers Heads (Q/KV) Context Length
Qwen3-235B-A22B 235B 22B 94 64 / 4 128K
Qwen3-30B-A3B 30B 3B 48 32 / 4 128K

Dense Models

Model Layers Heads (Q/KV) Tie Embedding Context Length
Qwen3-32B 64 64 / 8 No 128K
Qwen3-14B 40 40 / 8 No 128K
Qwen3-8B 36 32 / 8 No 128K
Qwen3-4B 36 32 / 8 Yes 32K
Qwen3-1.7B 28 16 / 8 Yes 32K
Qwen3-0.6B 28 16 / 8 Yes 32K

Hybrid Thinking and Budget Control

Qwen3 implements a dual-mode approach to inference, enabling a scalable and smooth correlation between the computational reasoning budget and performance improvements.

  • Thinking Mode: The model reasons step-by-step before delivering a final answer, optimized for complex tasks.
  • Non-Thinking Mode: The model provides near-instant responses for simpler queries where speed is prioritized over depth.

Users can control this behavior via the enable_thinking argument in the chat template or by using soft-switch tags like /think and /no_think within prompts to change modes dynamically during multi-turn conversations.

Pre-training and Dataset Expansion

Qwen3 was pre-trained on approximately 36 trillion tokens—nearly double the 18 trillion used for Qwen2.5—covering 119 languages and dialects. The data collection process included web data and PDF-like documents extracted via Qwen2.5-VL and refined by Qwen2.5. Synthetic data for math and code was generated using Qwen2.5-Math and Qwen2.5-Coder.

Pre-training occurred in three stages:

  1. S1: Basic language skills and general knowledge training on 30+ trillion tokens (4K context length).
  2. S2: Focus on STEM, coding, and reasoning tasks with an additional 5 trillion tokens.
  3. S3: Extension of context length to 32K using high-quality long-context data.

Post-training Pipeline

To achieve the hybrid thinking capability, Qwen3 underwent a four-stage post-training process:

  1. Long CoT Cold Start: Fine-tuning on diverse long chain-of-thought data across math, coding, and STEM.
  2. Reasoning-based RL: Scaling computational resources for reinforcement learning using rule-based rewards.
  3. Thinking Mode Fusion: Integrating non-thinking capabilities by fine-tuning on a mix of long CoT and standard instruction-tuning data.
  4. General RL: Applying RL across 20+ general-domain tasks to improve instruction following, formatting, and agentic capabilities.

Agentic Capabilities and Integration

Qwen3 is optimized for tool calling and supports the Model Context Protocol (MCP). For implementation, Qwen recommends using the Qwen-Agent framework, which simplifies tool-calling templates and parsers.

Deployment Recommendations

  • Serving Frameworks: SGLang (>=0.4.6.post1) or vLLM (>=0.8.4) for OpenAI-compatible API endpoints.
  • Local Usage: Ollama, LMStudio, MLX, llama.cpp, and KTransformers.

Sources