Qwen3.8-Flash-Next release notes / what's new

Qwen3.8-Flash-Next is a multimodal Mixture-of-Experts (MoE) model that serves as an architectural preview for the upcoming Qwen4 family. It introduces a hybrid Gated DeltaNet (GDN) and Qwen Sparse Attention (QSA) design to optimize computational efficiency and model capacity, reducing training costs to approximately 1/9th of those required for Qwen3.7-Plus while delivering superior performance in coding and office productivity tasks.

Model Architecture and Technical Innovations

Qwen3.8-Flash-Next implements systematic upgrades across four primary technical dimensions: attention, residual streams, embeddings, and optimization.

Hybrid Attention: GDN and QSA

To balance memory efficiency with precise retrieval, the model uses a hybrid architecture where three out of every four layers utilize Gated DeltaNet (GDN) to compress historical information into a fixed-size state. The remaining layer employs global attention enhanced by Qwen Sparse Attention (QSA).

QSA reduces the overhead of long-sequence computation by using a lightweight indexer to aggregate sequences into micro-blocks. It estimates importance at the block level and selects relevant regions for attention, reducing both the attention cost and the indexing overhead. At a 1-million-token context length, QSA's attention kernel achieves up to 7.6× speedups in prefill and 4.9× speedups in decode.

Gated Residual (GR)

To prevent the dilution of early features in deep networks, Gated Residual (GR) expands the traditional single residual stream into four parallel branches. A dynamic, element-wise gate controls the reading and writing of information across these branches, strengthening cross-layer information flow and improving training stability. The residual state also supports FP8 storage to further reduce memory-access overhead.

N-gram Embedding

To scale model capacity without increasing per-token computation, Qwen3.8-Flash-Next introduces N-gram Embedding. This method performs lookups based on the current token and several preceding tokens to represent common phrases and local patterns. The model includes 51B N-gram embedding parameters, which can be offloaded to host memory and asynchronously prefetched, ensuring they do not occupy permanent GPU memory.

Optimization via Muon

The model is trained using the Muon optimizer, specifically refined for orthogonalization accuracy and the splitting of fused parameter matrices. Muon is applied to two-dimensional linear maps (Attention, GDN, and MoE Experts), while AdamW is retained for embeddings, the MoE router, and low-rank parameters in GR. This co-design allows for larger learning rates and batch sizes, improving convergence efficiency.

Model Specifications and Capabilities

Qwen3.8-Flash-Next features a 125B-parameter main model with an additional 51B N-gram embeddings, activating 6B parameters per token.

Context Window and Performance

  • Context Support: Natively supports 262,144 tokens, extensible to 1,000,000 tokens via YaRN.
  • Efficiency: In serving scenarios with a 90% prefix cache hit rate, it achieves 8.6× the prefill throughput of Qwen3.7-Plus at 1M tokens.
  • Pricing: The production version (Qwen3.8-Flash) is priced at 0.16 USD per million input tokens and 0.47 USD per million output tokens.

Benchmarks

In base model comparisons, Qwen3.8-Flash-Next-Base outperformed Qwen3.8-27B-Base and Qwen3.7-Plus-Base on 8 of 14 benchmarks, including MMLU-Pro, SuperGPQA, BBH, GSM8K, EvalPlus, SWEBench-Pretrain, MGSM, and MMMLU.

For agentic and coding tasks, the model shows significant gains:

  • DeepSWE 1.1: 58.7 (compared to 16.5 for Qwen3.7-Plus).
  • SWE-bench Pro: 62.5 (compared to 55.8 for Qwen3.7-Plus).
  • CoWorkBench: 73.9 (compared to 65.1 for Qwen3.7-Plus).
  • LiveCodeBench v6: 91.9.

Multimodal and Vision-Language Performance

Qwen3.8-Flash-Next demonstrates strong capabilities in multimodal agentic intelligence and general perception:

  • Multimodal Tool Use (ClawEval-MM): 64.4 Pass@3.
  • Mobile Use (AndroidWorld): 84.5.
  • Computer Use (OSWorld 2.0): 19.4 Binary / 52.3 Partial.
  • Long Video Understanding (LVBench): 76.6.
  • Visual Math (MathVision): 95.7 with CI.

Integration and Availability

Weights are available on Hugging Face and ModelScope. The model is served via QwenCloud under the name qwen3.8-flash and supports OpenAI-compatible Chat Completions and Responses APIs, as well as an Anthropic-compatible interface for direct integration with tools like Claude Code.

Sources

Related