QwenLM/Qwen3.8-Flash-Next

What it solves

Qwen3.8-Flash-Next is a multimodal Mixture-of-Experts (MoE) model designed to maximize computational efficiency and training stability while maintaining high performance in coding and office tasks. It serves as an early architectural preview for the upcoming Qwen4 family, introducing a hybrid architecture that significantly reduces training and inference costs compared to previous versions like Qwen3.7-Plus.

How it works

The model employs a systematic upgrade across four key areas:

• Attention: Uses a hybrid GDN + QSA architecture. Gated DeltaNet (GDN) handles efficient history compression, while Qwen Sparse Attention (QSA) uses a lightweight indexer to select important context at a micro-block level, reducing costs for long sequences.
• Residual: Implements Gated Residual (GR), which expands the residual stream into four branches and uses a dynamic gate to control information flow and improve stability.
• Embedding: Utilizes N-gram Embedding, which uses local context to scale model capacity with minimal computation; these tables can be offloaded to host memory and prefetched asynchronously.
• Optimization: Employs the Muon optimizer, refined for orthogonalization accuracy and specific parameter splitting, with a refitted scaling law.

Who it’s for

This model is intended for developers and researchers who need a high-capacity model (125B parameters, 6B activated per token) that is cost-efficient to run and train, as well as the community wanting to examine the architectural changes leading into Qwen4.

Highlights

• Extreme Efficiency: Training costs are approximately 1/9th of Qwen3.7-Plus while delivering superior coding and office task performance.
• Multimodal MoE: Combines multimodal capabilities with a Mixture-of-Experts architecture.
• Hugging Face & ModelScope Support: Weights are available on both platforms for easy integration.
• Broad Deployment Support: Compatible with SGLang, vLLM, llama.cpp, and MLX for high-performance inference.