Qwen2.5-Turbo 1M Token Context Length Release

Qwen has released Qwen2.5-Turbo, extending the model's context window from 128k to 1 million tokens. This update enables the processing of massive datasets—equivalent to approximately 10 full-length novels or 30,000 lines of code—while achieving significant improvements in inference speed and cost-efficiency.

Expanded Context Capabilities

Qwen2.5-Turbo supports a context length of up to 1 million tokens, which translates to roughly 1 million English words or 1.5 million Chinese characters. This expansion allows for complex, high-volume data processing tasks, such as summarizing entire book trilogies or performing repository-level code analysis.

Long-Context Performance Benchmarks

The model demonstrates high precision in retrieving information from ultra-long sequences:

  • Passkey Retrieval: Achieves 100% accuracy in the 1M length Passkey Retrieval task.
  • RULER Benchmark: Scores 93.1, outperforming GPT-4 (91.6) and GLM4-9B-1M (89.9).
  • Complex Understanding: In benchmarks such as LV-Eval and LongbenchChat, Qwen2.5-Turbo surpasses GPT-4o-mini in most dimensions when processing contexts exceeding 128k tokens.

Short-Sequence Stability

A common issue with context extension is performance degradation on shorter sequences. Qwen2.5-Turbo is designed to maintain short-sequence capabilities on par with GPT-4o-mini, ensuring that increasing the context window does not compromise standard conversational or reasoning performance.

Inference Optimization and Cost Efficiency

Qwen2.5-Turbo introduces significant optimizations to address the computational demands of long-context processing through sparse attention mechanisms.

Speed and Efficiency

  • Reduced Latency: The use of sparse attention mechanisms has reduced the Time to First Token (TTFT) for 1M token contexts from 4.9 minutes to 68 seconds, representing a 4.3x speedup.
  • Computational Compression: Sparse attention compresses attention computation by approximately 12.5 times.

Pricing and Value

  • Cost-Effective Scaling: The model is priced at ¥0.3 per 1M tokens. At this price point, Qwen2.5-Turbo can process 3.6 times more tokens than GPT-4o-mini.

Implementation and Availability

Qwen2.5-Turbo is available via the Alibaba Cloud Model Studio API. The API is compatible with the OpenAI SDK, allowing for straightforward integration into existing Python workflows using the qwen-turbo-latest model identifier.

Developers can access the model through:

  • Alibaba Cloud Model Studio (API)
  • HuggingFace Demo
  • ModelScope Demo

Sources