Qwen 3.5‑397B‑A17B release: hybrid linear‑attention MoE model with 1 M token context and state‑of‑the‑art multimodal performance

TL;DR

Qwen 3.5‑397B‑A17B, the first open‑weight model of the Qwen 3.5 series, is a 397 billion‑parameter vision‑language model that activates only 17 billion parameters per forward pass, delivering state‑of‑the‑art performance on language, coding, reasoning, and multimodal benchmarks while keeping inference fast and cheap.

1. What was announced?

Qwen AI released Qwen 3.5‑397B‑A17B, the inaugural model of the Qwen 3.5 series. It is a native multimodal (vision‑language) model that combines linear attention via Gated Delta Networks with a sparse Mixture‑of‑Experts (MoE) architecture. Only 17 B of the 397 B parameters are active for any given input, giving an 8.6× (32 k context) to 19.0× (256 k context) speed‑up over the previous Qwen 3‑Max‑Base while preserving comparable accuracy.

2. Core technical innovations

2.1 Hybrid attention + sparse MoE

  • Gated Delta Network (GDN) provides linear‑time attention, drastically reducing the quadratic cost of classic self‑attention.
  • A sparse MoE layer routes each token to a small subset of experts, limiting the number of activated parameters to 17 B regardless of the total model size.
  • The combination yields high throughput (up to 19× faster than Qwen 3‑Max‑Base at 256 k context) without sacrificing the expressive power of a 397 B model.

2.2 Efficiency tricks

  • Multi‑token prediction and stability optimizations keep training stable at extreme scales.
  • FP8 pipeline (FP8 for activations, MoE routing, and GEMM) reduces activation memory by ~50 % and adds >10 % speed‑up, while critical layers keep BF16 precision.
  • Asynchronous RL framework supports all Qwen 3.5 sizes, enabling massive multi‑turn, multi‑modal reinforcement‑learning (RL) training with dynamic load balancing and low gradient staleness.

2.3 Language coverage

  • Vocabulary expanded to 250 k tokens.
  • Supported languages increased from 119 to 201 (including many dialects), improving global accessibility.

3. Benchmark performance

Qwen 3.5‑397B‑A17B was evaluated on a wide suite of frontier benchmarks. The table below highlights the most relevant scores (higher is better unless noted). All numbers are from the official Qwen 3.5 release.

Category Benchmark Qwen 3.5‑397B‑A17B Closest competitor
Knowledge MMLU‑Pro 87.4 GPT‑5.2 (89.5)
MMLU‑Redux 95.0 GPT‑5.2 (95.6)
C‑Eval 90.5 Claude 4.5 Opus (92.2)
Reasoning LiveCodeBench v6 87.7 GPT‑5.2 (84.8)
HMMT Feb 25 99.4 Claude 4.5 Opus (92.9)
Coding Code‑Interpreter (IFEval) 94.8 GPT‑5.2 (90.9)
Multimodal MMMU‑Pro 85.0 Gemini‑3 Pro (70.4)
MathVision 84.2 GPT‑5.2 (74.6)
Real‑world VQA (RealWorldQA) 81.0 Gemini‑3 Pro (77.0)
Agent / Tool use General Agent (BFCL‑V4) 63.1 Claude 4.5 Opus (77.5)
Tool Decathlon 43.8 GPT‑5.2 (43.5)
Search Agent (HLE w/ tool) 45.5 Gemini‑3 Pro (49.8)

Across the board, Qwen 3.5‑397B‑A17B matches or exceeds the strongest existing models on language, reasoning, coding, and multimodal tasks while using far fewer active parameters per inference step.

4. Multimodal capabilities

Modality Representative benchmark Qwen 3.5‑397B‑A17B score
STEM & Puzzle MMMU 85.0
Math‑oriented vision MathVista (mini) 90.1
General VQA RealWorldQA 81.0
Document understanding OmniDocBench 1.5 90.8
OCR / Text‑in‑image OCRBench 93.1
Spatial reasoning RefCOCO (avg) 87.8
Video understanding VideoMME (w/ sub.) 87.4

The model processes up to 1 M tokens (≈2 h of video) in a single forward pass, enabling end‑to‑end video‑to‑code, video summarisation, and multimodal planning.

5. System‑level infrastructure

  • Heterogeneous parallelism: vision and language pipelines use separate parallelism strategies, avoiding the inefficiencies of a monolithic approach.
  • Near‑100 % training throughput on mixed text‑image‑video data compared with a pure‑text baseline.
  • FP8‑enabled pipeline cuts activation memory by ~50 % and yields >10 % speed‑up, while preserving BF16 for numerically sensitive layers.
  • Scalable asynchronous RL engine – supports multi‑turn, multi‑modal environments, speculative decoding, rollout‑router replay, and fine‑grained fault recovery, delivering a 3×–5× end‑to‑end speed‑up.

6. How to use Qwen 3.5

6.1 Qwen Chat (interactive)

Three interaction modes are exposed:

  • Auto – adaptive reasoning with tool use (search, code interpreter).
  • Thinking – deep chain‑of‑thought for hard problems.
  • Fast – instant answers without tool calls.

6.2 Qwen 3.5‑Plus (hosted on Alibaba Cloud ModelStudio)

Enable advanced features via environment variables:

export DASHSCOPE_API_KEY=your_key
export DASHSCOPE_MODEL=qwen3.5-plus   # optional override
# Enable reasoning chain‑of‑thought
export ENABLE_THINKING=true
# Enable web search & code interpreter
export ENABLE_SEARCH=true

A minimal Python snippet:

from openai import OpenAI
import os
client = OpenAI(api_key=os.getenv("DASHSCOPE_API_KEY"),
                base_url=os.getenv("DASHSCOPE_BASE_URL",
                                   "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"))
resp = client.chat.completions.create(
    model=os.getenv("DASHSCOPE_MODEL", "qwen3.5-plus"),
    messages=[{"role": "user", "content": "Introduce Qwen 3.5."}],
    extra_body={"enable_thinking": True, "enable_search": False},
    stream=True)
for chunk in resp:
    print(chunk.choices[0].delta.get("content", ""), end="")

The same endpoint also powers Qwen Code, Qwen Visual Agents, and vibe‑coding experiences.

7. Implications and future directions

  • Native multimodal reasoning: By fusing vision early in the transformer stack, Qwen 3.5 can reason about images, videos, and text in a single pass, a decisive step toward general‑purpose AI agents.
  • Parameter‑efficient scaling: Activating only 4–5 % of the total parameters per token demonstrates that massive models need not be prohibitively expensive at inference time.
  • Reinforcement‑learning‑driven agents: The asynchronous RL framework makes it feasible to train agents that learn from long‑horizon, multi‑modal environments—paving the way for persistent, memory‑augmented assistants.
  • Broader accessibility: Supporting 201 languages and a 250 k token vocabulary lowers the barrier for non‑English developers and enterprises.
  • Roadmap: The Qwen team hints at a forthcoming technical report that will detail larger scaling experiments, more diverse RL environments, and deeper integration of external tools (e.g., retrieval, planning, and economic awareness).

8. Bottom line

Qwen 3.5‑397B‑A17B proves that massive multimodal models can be both powerful and efficient. Its hybrid linear‑attention + MoE design delivers near‑state‑of‑the‑art accuracy while keeping inference cost comparable to a 1 T‑parameter model. The open‑weight release, extensive benchmark suite, and ready‑to‑use cloud service make it a compelling foundation for the next generation of AI assistants, autonomous agents, and multimodal applications.

Key takeaways

  • 397 B total parameters, 17 B active per token → up to 19× faster than Qwen 3‑Max‑Base.
  • State‑of‑the‑art results on language, coding, reasoning, and vision tasks.
  • 1 M‑token context window, enabling long‑form video and document processing.
  • Open‑weight model plus hosted Qwen 3.5‑Plus for immediate production use.

References

  • Qwen AI blog post – Qwen 3.5: Towards Native Multimodal Agents (2026‑02‑14).
  • Official model cards on GitHub, Hugging Face, and ModelScope.
  • Benchmark suites: MMLU‑Pro, C‑Eval, LiveCodeBench v6, MMMU‑Pro, VideoMME, etc.

Sources