vLLM Day 0 Support for Qwen3.8-2.4T-A95B

vLLM now provides Day-0 support for Qwen3.8-2.4T-A95B, enabling the deployment of a Qwen-Max-class model with open weights. This release integrates the Qwen 3.5 architecture, allowing the model to run on vLLM out of the box without requiring architecture changes.

Model Architecture and Hardware Requirements

Qwen3.8-2.4T-A95B is a sparse Mixture-of-Experts (MoE) model with 2.4 trillion parameters and 512 experts. Its 92-layer hybrid backbone utilizes a specific attention mechanism: full attention is applied every 4th layer, while the remaining 69 layers utilize linear attention.

Hardware requirements for inference vary by precision:

  • Full Precision/FP8: Requires at least two NVIDIA B300 or AMD MI355X nodes.
  • FP4 Quantized: Can be run on a single node.

Precision and Quantization Options

Beyond the official BF16 and FP8 checkpoints, Inferact has released MXFP4 and NVFP4-quantized weights. These FP4 versions use Round-to-Nearest (RTN) quantization with activation calibration to enable 4-bit activations, specifically targeting routed experts and selected layers to reduce memory and bandwidth overhead.

Initial verification shows that FP4 quantization maintains accuracy. For example, on the GSM8K benchmark (strict/flexible), the NVFP4 version achieved 90.37% / 91.05%, compared to FP8's 89.61% / 90.52%. On AIME25 @3 (avg/pass), NVFP4 reached 92.22% / 96.67% against FP8's 87.78% / 93.33%.

Hardware-Specific Optimizations

To support a model of this scale, vLLM collaborated with NVIDIA and AMD to develop optimized kernels:

NVIDIA Platforms

NVIDIA and Inferact developed ultra-fast kernels for Dense GEMMs, MoE routing, Attention (GQA), and Linear Attention (Gated Delta Rule). The implementation includes new fused kernels to minimize communication overhead and a specific decomposition of work that combines Data Parallelism and Tensor Parallelism for Attention and Expert Parallelism for the MoE.

AMD Instinct GPUs

Acceleration is achieved via AITER-fused Gated DeltaNet decode, attention, and MoE kernels, which reduce data-movement and kernel-launch overhead. The shared-expert path utilizes optimized hipBLASLt GEMM kernels, while routed experts use AITER FusedMoE. Additionally, AMD Quark quantization support enables efficient MXFP4 deployment.

Deployment and Configuration

For optimal performance, the Qwen 3.8 model card recommends the following generation parameters:

  • Temperature: 1.0
  • Top_p: 0.95
  • Top_k: 20
  • Min_p: 0.0
  • Presence Penalty: 0.0
  • Repetition Penalty: 1.0

Because Qwen 3.8 is a reasoning model, users are advised to set a high max_tokens value (e.g., 128,000) to provide sufficient token budget for agentic workflows.

Quick Start Commands

NVFP4 Deployment:

vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \
  --linear-backend flashinfer_cutedsl \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

MXFP4 Deployment:

vllm serve Inferact/Qwen3.8-2.4T-A95B-MXFP4 \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp":"mtp","num_speculative_tokens":3}'

Sources

Related

  • Dispatch
    Qwen2.5-1M Release: Open‑Source 7B and 14B Models with 1M‑Token Context and vLLM‑Based Inference Framework

    Qwen releases open‑source Qwen2.5‑7B‑Instruct‑1M and Qwen2.5‑14B‑Instruct‑1M models that support up to 1 million token contexts, accompanied by an optimized vLLM‑based inference framework that delivers 3×–7× faster prefill and retains short‑task performance comparable to GPT‑4o‑mini.

  • Dispatch
    Qwen3.8-2.4T-A95B Release Notes

    Qwen3.8-2.4T-A95B is a massive open-weight model with 2.4 trillion total parameters and 95 billion active parameters, designed for high-performance coding, research, and agentic tasks.

  • Dispatch
    vLLM Qwen3.5 Performance Optimization

    vLLM has achieved over 25,000 total tokens per second (TPS) per GPU for Qwen3.5 on GB200 NVL72 systems through Blackwell-optimized kernels, hybrid cache state transfer, and async scheduling.

  • Dispatch
    vLLM TML Inkling Support

    vLLM has announced Day-0 support for TML Inkling, a 1T-parameter multimodal model, delivering up to 380 tok/s/user on 4 GB200 GPUs through specialized architectural optimizations.

  • Dispatch
    vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard

    vLLM has achieved top rankings on the Artificial Analysis leaderboard for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B by implementing aggressive kernel fusion and speculative decoding optimizations.