Meta Muse Spark Open 816B multimodal MoE model release

TL;DR

Meta unveiled Muse Spark Open, an open‑weight 816 B parameter multimodal mixture‑of‑experts (MoE) model that processes text, images, audio, and video, supports up to 1 M tokens of context, and is optimized for agentic coding and tool‑calling.

Overall Architecture and Scale

Key takeaway: Muse Spark Open combines a massive 816 B total parameter count with only ~42 B active parameters per token, thanks to a 256‑expert MoE layer that selects 8 experts for each token.

  • Model size: ~816 B total parameters; ~42 B active per token.
  • MoE configuration: 256 experts per MoE layer, 8 experts routed per token.
  • Layers: 62 transformer layers, hidden dimension 8192, grouped‑query attention (128 query heads, 16 key‑value heads, 64‑dim per head).
  • Context window: Up to 1 048 576 tokens, encompassing text, image, audio, and video tokens.
  • Checkpoint formats: BF16 (≈1.5 TB safetensors) and NVFP4 (4‑bit quantization for NVIDIA Blackwell GPUs).
  • Parameter distribution: Experts hold ~97 % of parameters; only a fraction is active per token, enabling inference speed comparable to a 42 B dense model while retaining the capacity of an 816 B model.

Multimodal Perception Encoders

Key takeaway: Separate encoders transform visual and audio inputs into token streams compatible with the language model.

Image & Video Encoder

  • Parameters: 1.8 B total, 50 layers, hidden size 1536, 16 attention heads.
  • Patch processing: Images split into 14×14 patches, each yielding a 4096‑dimensional feature.
  • Positional encoding: Learned absolute embeddings plus 2‑D RoPE.
  • Attention pattern: Full attention every fourth layer and the final layer; sparse attention elsewhere.
  • Video handling: Same encoder processes video frames; frames are down‑sampled (default 2 FPS, max 96 frames) and interleaved with timestamp tokens.

Audio Encoder

  • Tokenization rate: One audio token every 80 ms (12.5 tokens/s).
  • Architecture: Convolutional pyramid with patch embedder stages of sizes [16, 6, 5, 4]; each stage contains two 1‑D conv blocks (kernel 7, stride 1).
  • Final projection: RMSNorm → linear → RMSNorm.
  • Input requirements: Mono waveform at 24 kHz, padded to multiples of 1 920 samples.

Long‑Context Mechanics

Key takeaway: Grouped‑query attention keeps the KV cache manageable, requiring ~250 KB per token (≈266 GB for a full 1 M‑token context) in BF16.

  • KV cache size per token: 2 × 16 heads × 64 dimensions × 62 layers = 126 976 values.
  • Implication: Enables extremely long prompts that can include extensive codebases, multi‑modal documents, or browsing histories.

Inference Interfaces

Key takeaway: The model is exposed through multiple popular serving stacks, each supporting the full multimodal tokenization pipeline.

Transformers (HF AutoModelForMultimodalLM)

  • Use MuseSparkProcessor to combine tokenizer, image/video processors, and audio feature extractor.
  • Configure parallelism via DistributedConfig (tp_size for tensor parallelism, ep_size for expert parallelism).
  • Example code snippet shows loading, preparing a chat template, and generating up to 2000 new tokens.

Text‑Only, Image‑Text, Video, and Audio Inference

  • Text‑only: Reasoning effort levels (low/medium/high/xhigh) map to internal integers; reasoning cannot be disabled.
  • Image‑text: Processor expands image markers into patch tokens, preserving alignment with surrounding text.
  • Video: Frames are down‑sampled, timestamps inserted, and optional audio extraction (load_audio_from_video=True).
  • Audio: Processor handles resampling, padding, and token insertion automatically.

Tool Calling

  • Supports both text‑only and multimodal tool calls.
  • Uses a channel‑based chat template (Harmony) with distinct recipients (self for reasoning, commentary for tool‑call commentary, user for final answer).
  • Example demonstrates calling a weather API after seeing an image.

Serving Stack Integrations

Key takeaway: Muse Spark Open can be deployed with llama.cpp, SGLang, vLLM, and MLX, each offering quantized variants for different hardware.

  • llama.cpp: 2‑bit GGUF quant (Q2_K) runs at ~18 tokens/s on a Mac M3 Ultra (512 GB) with ample KV cache space.
  • SGLang: NVFP4 checkpoint runs on four GB300 GPUs; API separates reasoning_content from final content.
  • vLLM: NVFP4 checkpoint with expert parallelism; supports 128 K token context length.
  • MLX / oMLX: 3‑bit and 4‑bit quantizations (oQ3e, oQ4e) run on Apple Silicon; 3‑bit model consumes ~344 GB RAM, achieving 225 tok/s prefill and 21 tok/s generation (1024‑prompt, 128‑token generation).

Fine‑Tuning with TRL

Key takeaway: Full‑size checkpoint can be fine‑tuned via LoRA adapters while keeping all experts frozen, enabling efficient adaptation.

  • Parallelism: DistributedConfig(fsdp_size=64, ep_size=8) shards the model across 64 H100 GPUs and distributes experts into 8 groups.
  • LoRA targets: Attention projection matrices (q_proj, k_proj, v_proj, o_proj).
  • Training run: 500 steps on 8 × 8 H100 nodes (64 GPUs total), ~8 000 tokens/s, token accuracy improved from 72 % to 84 % and loss from 1.26 to 0.57.
  • Script highlights: packing=True merges examples into 2048‑token sequences; save_only_model=True stores only LoRA adapters.

Deployment on Hugging Face Inference Endpoints

Key takeaway: A single CLI command deploys the NVFP4 variant to an OpenAI‑compatible endpoint with optional 1 M‑token context.

hf endpoints catalog deploy --repo meta-models/Muse-Spark-Open-816B-A42B-NVFP4 --name muse-spark
  • Supports both /v1/chat/completions and /v1/responses APIs.
  • Works on 8 × RTX Pro 6000 (98 GB) for 131 072‑token context or 8 × H200 (141 GB) for the full 1 M‑token window.

Demonstrations of Agentic Multimodality

Key takeaway: The model excels at coding‑centric tasks that combine vision, audio, and interactive tool use.

  • 3D Asset Generator: Quantized model (2‑bit) drives a Pi coding agent to generate and iteratively refine Three.js assets, using visual feedback loops.
  • Browser Agent (OpenEnv + HF Sandbox): Muse Spark Open receives screenshots and audio prompts, decides actions, and interacts with a headless Chromium browser without any DOM information.
  • Call‑Your‑Browser Space: Voice‑driven product search on Amazon, showcasing multimodal reasoning, tool calling, and audio input handling.

Wrapping Up

Muse Spark Open is the largest open‑weight multimodal MoE model released to date, offering a unified interface for text, image, audio, and video inputs, a 1 M‑token context window, and built‑in agentic reasoning and tool‑calling capabilities. The model is available in BF16 and NVFP4 checkpoints, with numerous quantized variants for diverse hardware, and can be fine‑tuned via LoRA while keeping experts frozen. Researchers and developers can start experimenting immediately through Hugging Face Transformers, llama.cpp, SGLang, vLLM, MLX, or hosted Inference Endpoints.

Sources