Qwen2 Release Notes / What's New

Qwen has released Qwen2, an evolution of the Qwen1.5 series featuring five model sizes (0.5B, 1.5B, 7B, 57B-A14B, and 72B) designed to improve performance in coding, mathematics, and multilingual understanding. The release is characterized by the adoption of Group Query Attention (GQA) across all sizes to optimize inference speed and memory usage, and the expansion of training data to include 27 additional languages beyond English and Chinese.

Model Architecture and Specifications

Qwen2 introduces a range of base and instruction-tuned models with varying parameter counts and context window capabilities. A key architectural update is the universal implementation of Group Query Attention (GQA) across all model sizes, which reduces memory overhead and increases inference speed.

Model Parameters Non-Embedding Params GQA Tie Embedding Context Length
Qwen2-0.5B 0.49B 0.35B True True 32K
Qwen2-1.5B 1.54B 1.31B True True 32K
Qwen2-7B 7.07B 5.98B True False 128K
Qwen2-57B-A14B 57.41B 56.32B True False 64K
Qwen2-72B 72.71B 70.21B True False 128K

For smaller models (0.5B and 1.5B), Qwen employs tied embeddings to prevent large sparse embeddings from dominating the total parameter count.

Multilingual Expansion and Code-Switching

Qwen2 has been trained on 27 additional languages to bolster its global utility. This expansion covers several key regions:

  • Western Europe: German, French, Spanish, Portuguese, Italian, Dutch
  • Eastern & Central Europe: Russian, Czech, Polish
  • Middle East: Arabic, Persian, Hebrew, Turkish
  • Eastern Asia: Japanese, Korean
  • South-Eastern Asia: Vietnamese, Thai, Indonesian, Malay, Lao, Burmese, Cebuano, Khmer, Tagalog
  • Southern Asia: Hindi, Bengali, Urdu

Beyond adding languages, the team focused on improving the model's ability to handle "code-switching" (mixing multiple languages within a single prompt), resulting in a substantial reduction of associated errors in multilingual evaluations.

Performance Benchmarks

Large-Scale Model Performance (Qwen2-72B)

Qwen2-72B demonstrates superior performance over its predecessor, Qwen1.5-110B, despite having fewer parameters. In base model evaluations, Qwen2-72B outperforms Llama-3-70B across several metrics, including MMLU (84.2 vs 79.5) and HumanEval (64.6 vs 48.2).

Instruction-Tuned Model Capabilities

Qwen2-72B-Instruct shows significant improvements over Qwen1.5-72B-Chat and remains competitive with Llama-3-70B-Instruct. Key highlights include:

  • Coding and Math: Integration of CodeQwen1.5 data has improved programming proficiency. Qwen2-72B-Instruct achieves a 59.7 score on MATH, compared to 50.4 for Llama-3-70B-Instruct.
  • Reasoning: The model exhibits strong performance on Arena-Hard (48.1) and MT-Bench (9.12).

Small and Mid-Size Models

Qwen2-7B-Instruct outperforms SOTA models of similar sizes, particularly in coding and Chinese-related metrics. Qwen2-0.5B and 1.5B also show gains over previous versions, with Qwen2-1.5B-Instruct reaching 52.4 on MMLU.

Long Context Understanding

All instruction-tuned models were pretrained on 32K token contexts. Through the use of YARN and Dual Chunk Attention, the models have been extrapolated to handle longer sequences:

  • Qwen2-72B-Instruct and Qwen2-7B-Instruct: Support up to 128K tokens, demonstrating nearly flawless information extraction in "Needle in a Haystack" tests.
  • Qwen2-57B-A14B-Instruct: Supports up to 64K tokens.
  • Small Models (0.5B/1.5B): Support up to 32K tokens.

Safety and Multilingual Responsibility

Qwen2-72B-Instruct is designed to be helpful, honest, and harmless. In safety evaluations involving multilingual unsafe queries (Illegal Activity, Fraud, Pornography, and Privacy Violence), Qwen2-72B-Instruct performed comparably to GPT-4 and significantly outperformed Mistral-8x22B. For example, in the "Illegal Activity" category, Qwen2-72B-Instruct maintained a 0% harmful response rate across all tested languages, matching GPT-4.

Licensing and Availability

Qwen has updated the licensing for the series to encourage broader commercial adoption:

  • Apache 2.0: Applied to Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, and Qwen2-57B-A14B.
  • Qianwen License: Retained for Qwen2-72B and its instruction-tuned variants.

Models are available on Hugging Face and ModelScope.

Sources