vLLM Day 0 support for Mistral Large 4

vLLM has announced full day-0 support for Mistral Large 4, Mistral's latest frontier mixture-of-experts (MoE) multimodal model. This integration enables high-performance inference through a new model runner (MRV2) and includes support for advanced architectural features like cross-layer KV sharing, NoPE layers, and DSpark speculative decoding.

Model Architecture and Capabilities

Mistral Large 4 is a 72-layer MoE model with 128 routed experts (top-4 selection) and one shared expert. It utilizes a DeepSeek-style sparse attention stack, including Multi-Head Latent Attention (MLA) and a DSA sparse indexer. Key architectural highlights include:

  • Parameters: 1 trillion total parameters, with 49 billion active per token.
  • Multimodal: Integrated Pixtral vision encoder supporting multiple images per prompt.
  • Efficiency: Uses cross-layer KV sharing where nine NoPE layers read the latent cache from layer 35, significantly reducing memory overhead.
  • Quantization: Supports both FP8 (block-scaled) and NVFP4 formats, with the latter matching FP8 performance within 0.3 points on core benchmarks while providing 1.5x the KV cache capacity.

Performance Optimizations

The vLLM implementation introduces several optimizations to handle the model's scale and long-context requirements:

  • Decode Context Parallelism (DCP): Shards the latent KV cache across tensor-parallel ranks, allowing for massive increases in KV capacity—up to 18.88M tokens on TP8 FP8 configurations.
  • DSpark Speculative Decoding: A built-in drafter that uses sliding-window MLA layers to achieve ~2.9 tokens per step, delivering approximately 2x output throughput.
  • Disaggregated Serving (NIXL): Supports prefill/decode disaggregation, allowing independent scaling of compute-heavy prefill and memory-heavy decode nodes.
  • HiSparse: Enables host-memory sparse KV offload, now compatible with cross-layer KV sharing, allowing for extended context windows by keeping non-resident KV in pinned host memory.

Hardware Support

Beyond standard NVIDIA GPU deployments, the release includes day-0 support for AMD Instinct MI350 Series GPUs via ROCm. This includes:

  • MXFP4 Experts: Support for OCP MXFP4 quantization on experts, reducing memory footprint and enabling the 1T model to run on four MI350 GPUs.
  • AITER Integration: Utilization of AMD's AITER kernel library for fused MoE operations, MLA decode kernels, and optimized all-reduce operations.
  • Specialized Decode: A latency-optimized path for low-concurrency serving that fuses MoE layers into persistent kernel launches, reducing per-token latency significantly.

Deployment

Mistral Large 4 is available via the vllm/vllm-openai:mistral-large-4 Docker image. Users can configure the model using standard vLLM flags for tensor parallelism, speculative decoding, and tool-call parsing. The implementation also supports TPU v7x environments, leveraging vllm-torchtpu for sparse MLA and DSA indexer kernels.

Sources