Qwen3.8-Flash-Next Release Notes: A Preview of Qwen4 Architecture

Qwen3.8-Flash-Next is a multimodal Mixture-of-Experts (MoE) model that introduces a systemic architectural overhaul across attention, residual streams, embeddings, and optimization. Designed as an early preview of the Qwen4 architecture, it achieves superior performance in coding and office tasks compared to Qwen3.7-Plus while requiring only 1/9 of the training cost.

Architectural Innovations

Qwen3.8-Flash-Next implements four primary technical upgrades to improve computational efficiency and model capacity.

Hybrid Attention: GDN and QSA

To balance memory efficiency with precise retrieval, the model uses a hybrid architecture where three out of every four layers utilize Gated DeltaNet (GDN) to compress historical information into a fixed-size state. The remaining layers use global attention enhanced by Qwen Sparse Attention (QSA).

QSA reduces the overhead of long-sequence computation by using a lightweight indexer that aggregates sequences into micro-blocks. By estimating importance at the block level rather than the token level, QSA minimizes indexing costs. At a 1M-token context length, QSA achieves up to 7.6x speedups in prefill and 4.9x in decode. In scenarios with a 90% prefix cache hit rate, it delivers 8.6x the prefill throughput of Qwen3.7-Plus.

Gated Residual (GR) Streams

To prevent the dilution of early-layer features in deep networks, Gated Residual (GR) expands the traditional single residual stream into four parallel branches. A dynamic, element-wise gate controls how information is read from and written to these branches.

This design allows the model to maintain long-range pathways from early attention layers to deeper layers. Additionally, the GR mechanism suppresses activation outliers, improving training stability, and supports FP8 storage to reduce memory-access overhead.

N-gram Embedding for Scalable Capacity

Qwen3.8-Flash-Next incorporates N-gram Embedding, which performs lookups based on local context (the current token and preceding tokens) rather than a single token. This adds a "local-pattern memory" that increases model capacity with negligible per-token computation.

The model includes 51B N-gram embedding parameters in addition to the 125B main model parameters. To avoid GPU memory saturation, these parameters can be stored in host memory and asynchronously prefetched during model computation.

Optimization via Muon

The model is trained using the Muon optimizer, specifically refined for orthogonalization accuracy and the splitting of fused parameter matrices. Muon is applied to two-dimensional linear maps (Attention, GDN, and MoE Experts), while AdamW is retained for embeddings, the MoE router, and GR low-rank parameters.

Key optimization findings include:

• Scaling Law Refitting: The new architecture allows for larger learning rates and batch sizes.
• Batch Size Warmup: The team found that starting directly with the target batch size is more efficient, reducing optimizer steps by 18.8% compared to gradual warmup.

Model Specifications and Performance

Qwen3.8-Flash-Next features a 125B-parameter main model with 6B parameters activated per token. It natively supports a context window of 262,144 tokens, extensible to 1,000,000 tokens via YaRN.

Benchmarks

In language and coding tasks, Qwen3.8-Flash-Next outperforms several competitors and previous versions:

Benchmark	Qwen3.8-Flash-Next	Qwen3.8-27B	Qwen3.7-Plus	DeepSeek-V4-Flash
DeepSWE 1.1	58.7	42.2	16.5	54.4
SWE-bench Pro	62.5	61.7	55.8	56.0
CoWorkBench	73.9	70.7	65.1	45.1
GPQA Diamond	91.7	89.2	90.3	90.8
LiveCodeBench v6	91.9	90.3	89.6	90.6

In vision-language tasks, the model shows strong agentic capabilities, particularly in AndroidWorld (84.5) and OSWorld 2.0 (52.3 partial), significantly exceeding the performance of Qwen3.7-Plus.

Deployment and Pricing

Qwen3.8-Flash-Next is available as an open-weight model on Hugging Face and ModelScope. The production version, Qwen3.8-Flash, is served via QwenCloud with the following pricing:

• Input: 0.16 USD per million tokens
• Output: 0.47 USD per million tokens

Community Insights and Technical Discussion

Community feedback highlights both the efficiency gains and the hardware requirements of the new architecture:

• Hardware Constraints: Users noted that the 51B N-gram embedding parameters significantly increase the memory footprint. While the 6B active parameters help with memory bandwidth, the total size makes it challenging for users with less than 128GB of RAM. One user noted:

  "the 50B ngram sidecar makes it impossible... the new ngram architecture makes it pretty much unusable for regular folks who cant afford more than 32-64 GB ram."

• Local Execution: Early adopters reported successful execution on Ryzen 395 / Strix Halo hardware, achieving approximately 22 tokens/s. Some users have implemented tentative llama.cpp branches to enable local support.

• Performance vs. Cost: The model's ability to handle complex agentic tasks (such as code archaeology and regression fixing) at a very low cost was cited as a major advantage, with one user reporting a complex merge and fix costing only $0.45.

• Quantization: There is ongoing discussion regarding the effectiveness of different quantization levels (e.g., IQ4_XS), with some users observing a performance drop compared to the dense Qwen3.8-27B model when using highly compressed quants.