Qwen1.5 Release Notes / What's New

Qwen has released Qwen1.5, a series of open-source base and chat models across multiple sizes and an MoE version, designed to improve human preference alignment, multilingual performance, and the overall developer experience. This release is significant because it integrates the models directly into the Hugging Face transformers library, removing the need for custom remote code and expanding compatibility with major deployment and finetuning frameworks.

Model Availability and Ecosystem Integration

Qwen1.5 provides base and chat models in eight sizes: 0.5B, 1.8B, 4B, 7B, 14B, 32B, 72B, and 110B, alongside a Mixture-of-Experts (MoE) model. To support diverse deployment environments, the lab provides quantized versions including Int4 and Int8 GPTQ, AWQ, and GGUF formats.

Developer Experience Improvements

The primary technical shift in Qwen1.5 is the merge of its code into Hugging Face transformers (version 4.37.0 and above). This allows developers to load models without setting trust_remote_code=True.

Qwen1.5 is also natively supported by the following ecosystems:

  • Deployment: vLLM (>=0.3.0) and SGLang (>=0.1.11).
  • Quantization: AutoAWQ and AutoGPTQ.
  • Finetuning: Axolotl and LLaMA-Factory.
  • Local Inference: llama.cpp, Ollama, and LMStudio.
  • API Services: DashScope and together.ai.

Technical Capabilities and Performance

All Qwen1.5 models uniformly support a context length of up to 32,768 tokens. The series demonstrates strong performance across language understanding, coding, reasoning, and mathematics.

Basic Benchmarks

Qwen1.5-72B outperforms Llama2-70B across all evaluated benchmarks, including MMLU, C-Eval, GSM8K, MATH, HumanEval, MBPP, and BBH. For small-scale models (under 7B parameters), Qwen1.5-0.5B, 1.8B, and 4B are positioned as highly competitive alternatives to other community small language models (SLMs).

Human Preference Alignment

Qwen1.5 chat models were aligned using Direct Policy Optimization (DPO) and Proximal Policy Optimization (PPO). According to MT-Bench and AlpacaEval 2.0 results, Qwen1.5-72B-Chat surpasses Claude-2.1, GPT-3.5-Turbo-0613, Mixtral-8x7b-instruct, and TULU 2 DPO 70B, performing on par with Mistral Medium.

Multilingual Proficiency

The base models were evaluated across 12 languages from Europe, East Asia, and Southeast Asia. Performance was measured across four dimensions: Exams, Understanding, Translation, and Math. Qwen1.5-72B demonstrates strong capabilities in languages including Arabic, Spanish, French, Japanese, Korean, and Thai.

Long Context Performance

Evaluated on the L-Eval benchmark, Qwen1.5-72B-Chat significantly outperforms GPT3.5-turbo-16k and closely follows GPT4-32k. Even the smaller Qwen1.5-7B-Chat is competitive with GPT-3.5 on four out of five L-Eval tasks.

External System Integration and Agentic Capabilities

Qwen1.5-Chat models are designed to integrate with external knowledge via Retrieval-Augmented Generation (RAG) and function calling for AI agent workflows.

RAG and Tool Use

  • RAG: On the RGB benchmark, Qwen1.5-72B-Chat shows strong performance in both English and Chinese, particularly in rejection and integration tasks.
  • Tool Selection: In tool-use benchmarks, Qwen1.5-72B-Chat nears GPT-4's performance in tool selection and input accuracy.
  • Agent Performance: On the T-Eval benchmark, Qwen1.5-72B-Chat achieves high scores in English and Chinese, particularly in planning and retrieval.

Code Interpreter

While Qwen1.5-72B-Chat performs well in general code interpreter tasks (87.9% accuracy), it lags behind GPT-4 in specialized math problem-solving and visualization tasks. The Qwen team has identified enhancing coding capabilities during pre-training and alignment as a priority for future versions.

Sources