Introducing Qwen: A Comprehensive LLM and LMM Framework

Qwen is a project aimed at achieving Artificial General Intelligence (AGI) that integrates large language models (LLMs) and large multimodal models (LMMs). The ecosystem includes base language models, chat-optimized versions (Qwen-Chat), domain-specific models for coding (Code-Qwen) and mathematics (Math-Qwen), and multimodal extensions for vision (Qwen-VL) and audio (Qwen-Audio).

Base Model Architecture and Scaling

Qwen is a Transformer-based language model pretrained using next-token prediction. The development focuses on scaling both model size and data volume to ensure stability and performance.

Model Variants and Specifications

Four of the five developed models are open-sourced, with specifications as follows:

Model	Release Date	Max Length	System Prompt Enhancement	Pretrained Tokens	Min GPU Memory (Q-Lora Finetuning)	Min GPU Usage (2048 Tokens Int4)	Tool Usage
Qwen-1.8B	23.11.30	32K	✓	2.2T	5.8GB	2.9GB	✓
Qwen-7B	23.08.03	32K	✐	2.4T	11.5GB	8.2GB	✓
Qwen-14B	23.09.25	8K	✐	3.0T	18.7GB	13.0GB	✓
Qwen-72B	23.11.30	32K	✓	2.2T	61.4GB	48.9GB	✓

Multilingual Capabilities and Tokenization

Qwen is designed as a multilingual model, demonstrating strong capabilities in English and Chinese, with additional proficiency in Spanish, French, and Japanese. This is achieved through a high-efficiency tokenizer that provides a high compression rate across multiple languages, optimizing the encoding of information.

Context Length and Extrapolation

Most open-sourced Qwen models support a context length of 32K tokens. This was achieved through continual pretraining with longer context lengths and an increased base value for Rotary Positional Embeddings (RoPE). Evaluation via L-Eval and "Needle in a Haystack" tests confirms the model's ability to handle long-context windows.

In a benchmark comparison, Qwen-72B-Chat (32K) achieved an average score of 62.30, outperforming ChatGPT-3.5-16k (60.73) in several categories, including TOEFL (86.24 vs 78.43) and QuALITY (77.22 vs 61.38).

Alignment and Post-Training

Qwen utilizes a two-stage alignment process consisting of Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF).

Supervised Fine-Tuning (SFT)

Alignment begins with SFT, where the team focuses on the diversity and complexity of the data, utilizing datasets such as instag and tulu 2. Quality is maintained through a combination of manual checking and automatic evaluation.

Reinforcement Learning from Human Feedback (RLHF)

Building upon the SFT model, Qwen employs PPO-based RLHF. To address training instabilities, the team developed a reliable reward model pretrained on large-scale comparison data and finetuned on high-quality, carefully labeled comparison data. The resulting RLHF models exhibit higher creativity and better instruction-following capabilities compared to SFT-only models.

Tool Use and Agentic Capabilities

Qwen is designed to function as an agent by generating thoughts and actions based on the ReAct format. This allows the model to use tools it has not seen during training by understanding instructions and demonstrations through in-context learning.

Supported Frameworks

Function Calling: Direct execution of specific functions.
Code Interpreter: Used for complex data analysis.
Hugging Face Agent: Integration with various AI models for diverse outputs, such as image generation.

AgentFabric

Following the concept of GPTs, Qwen has introduced AgentFabric, a framework that allows users to create specialized AI agents through a simple chat-based configuration process.