Qwen1.5-MoE-A2.7B Release Notes
Qwen has released Qwen1.5-MoE-A2.7B, a small Mixture-of-Experts (MoE) model that matches the performance of state-of-the-art 7B models, such as Mistral 7B and Qwen1.5-7B, while utilizing only 2.7 billion activated parameters. This architecture achieves a 75% reduction in training expenses and increases inference speed by 1.74x compared to Qwen1.5-7B.
MoE Architectural Enhancements
Qwen1.5-MoE-A2.7B utilizes a specialized MoE architecture that improves upon the standard top-2 gating strategy used in models like Mixtral. The architecture incorporates three primary modifications:
Fine-grained Experts
Instead of simply replicating a Feed-Forward Network (FFN) layer to create experts, Qwen partitions a single FFN into several segments. This approach creates a larger number of experts without increasing the total parameter count. The model utilizes 64 experts, an 8-time increase over the conventional 8-expert MoE setup.
Upcycling Initialization
To avoid the inefficiencies of training from scratch, the model was initialized by repurposing the existing Qwen-1.8B model. The researchers found that introducing randomness during this "upcycling" initialization stage significantly accelerated convergence and improved overall performance during pre-training.
Shared and Routing Experts
The routing mechanism employs a generalized approach combining shared and routing-specific experts. Qwen1.5-MoE-A2.7B uses 4 shared experts that are always activated, alongside 60 routing experts, of which 4 are activated per token. This configuration provides greater flexibility and efficiency than conventional MoE routing.
Performance Benchmarks
Qwen1.5-MoE-A2.7B demonstrates competitive performance across language understanding, mathematics, and coding benchmarks, performing similarly to dense 7B models.
| Model | MMLU | GSM8K | HumanEval | Multilingual | MT-Bench |
|---|---|---|---|---|---|
| Mistral-7B | 64.1 | 47.5 | 27.4 | 40.0 | 7.60 |
| Gemma-7B | 64.6 | 50.9 | 32.3 | - | - |
| Qwen1.5-7B | 61.0 | 62.5 | 36.0 | 45.2 | 7.60 |
| DeepSeekMoE 16B | 45.0 | 18.8 | 26.8 | - | 6.93 |
| Qwen1.5-MoE-A2.7B | 62.5 | 61.5 | 34.2 | 40.8 | 7.17 |
While the base model shows parity with 7B models, the team noted that there is still untapped potential for enhancing the chat model's performance through refined fine-tuning strategies.
Training and Inference Efficiency
The MoE architecture significantly reduces computational overhead compared to dense models by activating only a fraction of its total parameters.
Parameter Comparison
Qwen1.5-MoE-A2.7B contains 14.3 billion total parameters, but only 2.7 billion are activated during a forward pass. Its non-embedding parameter count (2.0 billion) is approximately one-third that of Qwen1.5-7B (6.4 billion).
Cost and Speed Gains
- Training Costs: The model achieved a 75% reduction in training expenses compared to Qwen1.5-7B, partly because upcycling eliminated the need to train on the same volume of tokens as the original model.
- Inference Speed: Tested on a single NVIDIA A100-80G GPU using vLLM (with 1000 input and 1000 output tokens), Qwen1.5-MoE-A2.7B-Chat achieved a throughput of 2.01 requests per second and 4010.27 tokens per second (TPS). This is 1.74x faster than Qwen1.5-7B-Chat, which recorded 1.15 throughput and 2298.89 TPS.
Deployment and Integration
Qwen1.5-MoE is integrated with the Hugging Face transformers library and vLLM. Because the qwen2_moe implementation is not yet in the main pip/conda release of transformers, users must install the library from source:
git clone https://github.com/huggingface/transformers
cd transformers
pip install -e .
For quantized deployment, the Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4 model is available, though AWQ is not currently supported.