vLLM Day 0 Support for Muse Spark Open: 816B-A42B MoE, DSpark Speculative Decoding, and LoRA RL

vLLM and Meta Superintelligence Labs shipped day 0 support for Muse Spark Open, an open-weight sparse mixture-of-experts model with 816B total parameters and 42B active per token, built for long-horizon agentic coding workflows. The release matters because it puts the most capable US open-weights model to date — the model Muse Glimmer 30B was distilled from — behind a production serving stack on Nvidia, AMD, and TPU hardware on the same day the weights landed.

What Muse Spark Open is

Muse Spark Open is an open-weight sparse mixture-of-experts (MoE) model designed for long-horizon agentic coding workflows. vLLM describes it as the most capable US open-weights model to date, and notes that Muse Glimmer 30B was distilled from it.

The headline specifications are:

  • Architecture: 816B total parameters, 42B active per token, with a 1M context window.
  • Quantization: Meta released the model in both BF16 and NVFP4. MXFP4 weights were released by AMD.
  • Multimodal: Native image and video perception through the same Perception Encoder used by Muse Glimmer, with additional audio support.
  • Parallelism: Tensor Parallelism and Pipeline Parallelism on both Nvidia and AMD. Expert Parallelism (EP), TEP, DEP, and PD disaggregation on Nvidia. Initial support on TPU is also available.
  • Speculative decoding: A DSpark draft model trained by the vLLM team, supported on both CUDA and ROCm, tested on the SPEED bench.

How to run it

Prebuilt images are available for both vendors:

  • Nvidia: vllm/vllm-openai:muse-spark-open
  • AMD: vllm/vllm-openai-rocm:muse-spark-open

The model will only be available in the nightly PyPI wheel after the code in PR #61879 is reviewed and merged, which vLLM says usually takes about a week.

The recommended serving command uses the Rust frontend and enables expert parallelism, a 1M-token max model length, the Muse reasoning and tool-call parsers, and DSpark speculative decoding:

# Rust frontend is recommended
export VLLM_USE_RUST_FRONTEND=1

vllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --max-model-len 1048576 \
    --reasoning-parser muse \
    --tool-call-parser muse \
    --enable-auto-tool-choice \
    --speculative-config '{"method":"dspark","model": "Inferact/Muse-Spark-Open-DSpark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'

A plain-text request can set reasoning_effort to xhigh:

curl http://localhost:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
      "messages": [{"role": "user", "content": "Refactor this module."}],
      "reasoning_effort": "xhigh"
    }'

Multimodal requests accept image, video, and audio content parts in a single message:

curl http://localhost:8000/v1/chat/completions \
   -H 'Content-Type: application/json' \
   -d '{
     "model": "meta-models/Muse-Spark-Open-816B-A42B",
     "messages": [{"role": "user", "content": [
       {"type": "image_url", "image_url": {"url": "https://example.com/screenshot.png"}},
       {"type": "video_url", "video_url": {"url": "https://example.com/demo.mp4"}},
       {"type": "audio_url", "audio_url": {"url": "https://example.com/clip.wav"}},
       {"type": "text",      "text": "What is happening here, and when?"}
     ]}]
   }'

Model architecture: where the 816B parameters live

Muse Spark Open is a mixture-of-experts model with 816B total parameters including the text decoder, plus a 2B vision encoder and a small audio encoder alongside it. The decoder has 62 layers, and every one of them is a mixture-of-experts layer.

The weight budget breaks down as follows:

part size
256 experts per layer 799 B
attention 13.5 B
MoE latent projections 4.2 B
embedding and output layer 3.3 B
vision tower (separate) 1.9 B

The architecture is similar to Muse Glimmer, with three design choices worth understanding.

Sandwich normalization

Most decoders place one normalization layer before each block, giving x + F(norm(x)). Muse Spark Open, like Muse Glimmer, places one normalization before and one after the block — "like the two slices of a sandwich with the block sitting in between" — giving x + post_norm(F(norm(x))).

Latent MoE

Every MoE layer carries two extra matrices shared by all 256 experts in that layer. The code calls them the pooler and unpooler, though neither is pooling in the usual sense; both are learned projections that squeeze and expand dimensions. The pooler is an 8192 × 4096 matrix that halves each token's width before it reaches the experts, and the unpooler restores the full width afterwards. The experts therefore work entirely at 4096 rather than 8192, which effectively halves the number of parameters required in the two expert projection layers.

Hybrid sliding-window and full attention

Layers alternate between full attention and Sliding Window Attention (SWA) on a repeating cycle of 4: three layers attend to a 2048-token window and the fourth looks at the whole context. The cycle is anchored to the end of the stack so the final layer is always a full attention layer. Across 62 layers, that yields 46 narrow layers and 16 wide ones, where the narrow layers only hold recent KV. At 1M tokens, this brings the cache to just over one quarter of what all-full attention would need — which is what makes the 1M context length affordable.

Vision and audio encoders

Muse Spark Open uses the same Perception Encoder as Muse Glimmer, a 2B-parameter tower that reads images at their native resolution rather than resizing them to a square. Videos run through the same encoder: frames are sampled at load time as a series of sampled video frames with the timestamp tokenized in the prompt.

Audio is read as a raw waveform at 24 kHz, and every 1920 samples — 80 ms — becomes one token through a series of transformations. The whole audio encoder is just 118M parameters. The video's audio track is not processed: to give the model both the pictures and the sound of a clip, send the video and the extracted audio as two separate content parts in the same request.

DSpark speculative decoding

To accelerate Muse Spark Open inference on vLLM, the team trained a speculator using DSpark, a speculative decoding algorithm that uses block diffusion to propose up to seven draft tokens in a single pass. Confidence-based scheduling helps sustain significant speedups even at large batch sizes, where speculative decoding traditionally offers diminishing returns.

The speculator also uses Muse Spark Open's sliding-window attention to keep drafting lightweight even with a 1M-token context. Together, these choices make it suited to production workloads spanning a wide range of batch sizes and context lengths.

DSpark is enabled at server start with:

--speculative-config '{"method":"dspark","model": "Inferact/Muse-Spark-Open-DSpark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'

Hardware backends: Nvidia, AMD, and TPU

Nvidia

The Nvidia team improved Muse Spark Open inference across FlashInfer and vLLM through fused normalization, gating, and residual kernels; fused MoE finalization with tensor-parallel communication and RMSNorm; a new low-latency BF16 GEMM implementation and reduced compilation overhead; TEP pooler/unpooler sharding with fused MNNVL collectives; TF32 router GEMM fed by the norm's FP32 output; and improvements to data movement, hardware support, and runtime reliability.

AMD

With support from the AMD team, Muse Spark Open runs on AMD Instinct MI355X GPUs in both FP8 and MXFP4 quantized formats, providing improved inference performance and deployment efficiency. ROCm currently supports both Tensor Parallelism and Pipeline Parallelism.

TPU

vLLM added Muse Spark Open support on TPU on top of TorchTPU; code access requires contacting vLLM or Google. The team also built megakernels for Muse Spark Open. With DSpark speculative decoding at an acceptance length of 6, they reach over 650 tokens/sec on 4 TPU v7 chips and over 960 tokens/sec on 8 chips. Without speculative decoding, they still deliver over 270 and 350 tokens/sec respectively.

Reinforcement learning in vLLM

vLLM supports LoRA RL on Muse Spark Open from day 0, serving as the rollout engine with Verl FSDP2 as the trainer. vLLM serves the original NVFP4 checkpoint with the LoRA adapter applied on top. The trainer runs on a BF16 dequantization of the same weights and updates only the adapter (rank 32, alpha 64), which covers attention, the MoE latent projections, and all routed experts. It also replays the rollout's MoE routing (R3), and only the adapter is synced back to vLLM after each step.

The team trained with GRPO on DAPO-Math-17K for 150 steps on 16 GB300 GPUs, sampling 8 responses for each of 32 prompts per step with up to 4K tokens per response. Accuracy on AIME-2024 (mean@8) went from 0.56 before training to 0.85 by the end of the run. Throughout the run, the k3 KL between the trainer's and vLLM's log-probabilities held near 2×10⁻⁴. More details are available in the Verl PR #8257. In the vLLM ecosystem, Titan RL also uses vLLM as its rollout engine to provide day-0 RL support.

Who built it

vLLM credits Meta Superintelligence Labs and the vLLM community for the day 0 support, Inferact for training the DSpark speculator and the TPU integration, and NVIDIA and AMD for hardware optimizations.

Contributors listed in the announcement:

  • Inferact: Jie Li, Zifeng Mo, Zixi Qi, Jeff Ma, Ziqi Zhao, Dakai An, Ao Shen
  • Meta: Beto de Paola, Sidhartha Reddy Potu
  • Nvidia: Cyrus Chang, Duncan Moss, Marek Pikulski, Seong Hee Lee
  • AMD: Andy Luo, Hongxia Yang, Haichen Zhang

Why this release matters

Day 0 support means the serving path, quantization formats, parallelism strategies, speculative decoding, and RL rollout integration all exist at the moment the weights are published — not weeks later. For an 816B-parameter MoE with a 1M-token context window, that combination of hybrid attention, latent MoE compression, and DSpark drafting is what makes the model practical to deploy rather than merely available. The simultaneous Nvidia, AMD, and TPU coverage also means the model is not locked to a single accelerator vendor.

Sources