Qwen2.5-1M Release: Open‑Source 7B and 14B Models with 1M‑Token Context and vLLM‑Based Inference Framework

Overview

Qwen announces the release of two open‑source instruction‑tuned models, Qwen2.5‑7B‑Instruct‑1M and Qwen2.5‑14B‑Instruct‑1M, each capable of processing contexts up to one million tokens, together with a fully open‑sourced inference framework built on vLLM that accelerates long‑context prefill.

Model Releases

The release provides two new checkpoints: Qwen2.5‑7B‑Instruct‑1M and Qwen2.5‑14B‑Instruct‑1M. These are the first Qwen models opened to the public with a 1M‑token context length.

Long‑Context Performance

Qwen2.5‑1M series models accurately retrieve hidden information from documents containing up to 1M tokens in the Passkey Retrieval task, with only minor errors observed in the 7B variant. On more complex long‑context benchmarks such as RULER, LV‑Eval and LongbenchChat, the models significantly outperform their 128K counterparts for sequences exceeding 64K tokens. The Qwen2.5‑14B‑Instruct‑1M model not only beats Qwen2.5‑Turbo but also consistently outperforms GPT‑4o‑mini across multiple datasets, offering a robust open‑source alternative for long‑context tasks.

Short‑Context Performance

Both Qwen2.5‑7B‑Instruct‑1M and Qwen2.5‑14B‑Instruct‑1M maintain short‑text task performance that is similar to their 128K versions, ensuring that fundamental capabilities are not compromised by the addition of long‑sequence processing. Compared to GPT‑4o‑mini, the Qwen2.5‑14B‑Instruct‑1M and Qwen2.5‑Turbo achieve similar performance on short text tasks while supporting a context length that is eight times longer.

Key Techniques

Long‑Context Training

Training follows a progressive schedule: starting from a 4K‑token checkpoint, the context length is increased to 256K tokens during pretraining using an adjusted RoPE base raised from 10,000 to 10,000,000. Supervised fine‑tuning is split into two stages—first on short instructions up to 32K tokens, then on a mix of short and long instructions up to 256K tokens—while reinforcement learning is performed on short texts up to 8K tokens. The resulting instruction‑tuned model handles 256K tokens; length extrapolation via Dual Chunk Attention (DCA) extends this to 1M tokens without additional training.

Dual Chunk Attention

DCA remaps relative positions to smaller values, preventing the model‑unseen during attention method allows models trained on smaller values, avoiding large unseen distances that cause RoPE‑based degradation. Ablation shows that even models trained on only 32K tokens achieve near‑perfect passkey retrieval accuracy at 1M tokens when DCA is applied.

Sparse Attention

The inference framework integrates a sparse attention mechanism based on MInference with several improvements: chunked prefill (chunk size 32,768 tokens) reduces activation VRAM usage by 96.7%; DCA is combined with MInference to boost efficiency and accuracy; a sparsity‑refinement method tailors the sparsification configuration for sequences up to 1M tokens, limiting accuracy loss; additional kernel and pipeline optimizations are applied. These enhancements yield a 3.2× to 6.7× acceleration in prefill speed for 1M‑token sequences across model sizes and GPU devices, consistent with the reported 3×–7× speedup.

Deploy Qwen2.5‑1M Models Locally

System Preparation

For optimal performance, use GPUs with Ampere or Hopper architecture. Required software: CUDA 12.1 or 12.3, Python 3.9–3.12. VRAM needed for 1M‑token processing: at least 120 GB total for the 7B model and at least 320 GB total for the 14B model (across GPUs). Lower VRAM limits the model to shorter tasks.

Install Dependencies

Clone the custom vLLM branch and install it:

git clone -b dev/dual-chunk-attn git@github.com:QwenLM/vllm.git
cd vllm
pip install -e . -v

Launch OpenAI‑Compatible API Service

Start the service with a command such as:

vllm serve Qwen/Qwen2.5-7B-Instruct-1M \
  --tensor-parallel-size 4 \
  --max-model-len 1010000 \
  --enable-chunked-prefill --max-num-batched-tokens 131072 \
  --enforce-eager \
  --max-num-seqs 1

Parameter meanings: --tensor-parallel-size equals the number of GPUs (max 4 for 7B, max 8 for 14B); --max-model-len sets the maximum input length; --max-num-batched-tokens defines the chunk size for chunked prefill (recommended 131,072); --max-num-seqs limits concurrent sequences.

Interact with the Model

Example using curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct-1M",
    "messages": [{"role": "user", "content": "Tell me something about large language models." }],
    "temperature": 0.7,
    "top_p": 0.8,
    "repetition_penalty": 1.05,
    "max_tokens": 512
  }'

Example using Python with the OpenAI client:

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)

prompt = ("There is an important info hidden inside a lot of irrelevant text. " +
          "Find it and memorize it. I will quiz you about the important information there.\n\n" +
          "The pass key is 28884. Remember it. 28884 is the pass key.\n" +
          "+
          "The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. " * 800 +
          "\nWhat is the pass key?
        )

chat_response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct-1M",
    messages=[{"role": "user", "content": prompt}],
    temperature=0,
)
print("Chat response:" + chat_response.choices[0].message.content)

Other options include the Qwen‑Agent framework for PDF reading and specialized tasks.

What’s Next

Qwen aims to improve long‑context models further by researching more efficient training methods, model architectures, and inference techniques to make them deployable in resource‑constrained environments while preserving strong performance on both short and long tasks.

Sources