Qwen1.5-110B release notes / what's new
Qwen has released Qwen1.5-110B, the first model in the Qwen1.5 series to exceed 100 billion parameters. This release demonstrates that increasing model size continues to provide performance gains in base capabilities and chat-based interactions, positioning the model as a competitive alternative to other large-scale open-source models like Llama-3-70B.
Model Architecture and Technical Specifications
Qwen1.5-110B utilizes a Transformer decoder architecture similar to other models in the Qwen1.5 series. Key technical specifications include:
- Grouped Query Attention (GQA): The model incorporates GQA to ensure efficiency during model serving.
- Context Window: It supports a context length of 32K tokens.
- Multilingual Support: The model maintains multilingual capabilities, supporting a wide range of languages including English, Chinese, French, Spanish, German, Russian, Korean, Japanese, Vietnamese, and Arabic.
Base Model Performance
Qwen1.5-110B is competitive with Meta-Llama3-70B and Mixtral-8x22B across several benchmarks. The Qwen team notes that performance improvements over the Qwen1.5-72B model were achieved by increasing the model size without drastically changing the pretraining and post-training recipes.
| Benchmark | Qwen1.5-110B | Qwen1.5-72B | Llama-3-70B | Mixtral-8x22B |
|---|---|---|---|---|
| MMLU | 80.4 | 77.5 | 79.5 | 77.8 |
| TheoremQA | 34.9 | 29.3 | 32.0 | 35.9 |
| GPQA | 35.9 | 36.3 | 36.4 | 34.3 |
| Hellaswag | 87.5 | 86.0 | 88.0 | 88.7 |
| BBH | 74.8 | 65.5 | 76.6 | 69.2 |
| ARC-C | 69.6 | 65.9 | 68.8 | 70.7 |
| GSM8K | 85.4 | 79.5 | 79.2 | 78.6 |
| MATH | 49.6 | 34.1 | 41.0 | 41.7 |
| HumanEval | 52.4 | 41.5 | 45.7 | 45.1 |
| MBPP | 58.1 | 53.4 | 55.1 | 71.2 |
Chat Model Evaluation
In chat-specific evaluations, Qwen1.5-110B-Chat shows significant improvement over the Qwen1.5-72B-Chat model. It outperforms Llama-3-70B-Instruct in both MT-Bench and AlpacaEval 2.0.
| Model | MT-Bench (Avg. Score) | AlpacaEval 2.0 (LC Win Rate) |
|---|---|---|
| Llama-3-70B-Instruct | 8.85 | 34.40 |
| Qwen1.5-72B-Chat | 8.61 | 36.60 |
| Qwen1.5-110B-Chat | 8.88 | 43.90 |
Implications for Model Scaling
The release of Qwen1.5-110B indicates that there is still significant room for performance gains through model size scaling. While the Qwen team acknowledges the importance of data scaling as demonstrated by Llama-3, they intend to pursue a strategy of scaling both data and model size for future releases, including the upcoming Qwen2.