Apple MLX Distributed Inference on Mac Studio: Muse Spark Open 816B Runs at 36+ Tokens/s Across Four Macs

Apple's ML Research team published a post on local and distributed large LLM inference with MLX and RDMA on Mac Studio. The headline result: Muse Spark Open, a nearly trillion-parameter model, generates over 36 tokens per second on a cluster of four Mac Studios with M5 Ultra — well above average human reading speed — and the distributed features rely on low-latency RDMA over Thunderbolt 5 introduced in macOS 26.2 and MLX v0.30.1 or later.

The post describes how MLX, Apple's open-source array framework for Apple silicon, lets researchers run frontier-class models locally in a private environment by exploiting unified memory. It follows an earlier Apple ML post on the M5 architecture and its GPU Neural Accelerators.

The three models Apple benchmarked

Apple evaluated three frontier-class models spanning a wide size range, using the MLX LM package and the mlx_lm.benchmark tool. Muse Spark Open is a mixture-of-experts (MoE) model that keeps only a fraction of its experts — and therefore its parameters — active at inference, reducing compute. Mistral Medium 3.5 and Qwen 3.8 are dense architectures, where all parameters are active at inference.

Model Total Parameters Active Parameters Native Precision Evaluation Precision
Muse Spark Open 816B 42B BF16 NVFP4
Mistral Medium 3.5 128B 128B FP8 FP8
Qwen 3.8 27B 27B BF16 BF16

Mistral Medium 3.5 and Qwen 3.8 were evaluated in their native floating-point precision. Muse Spark Open was evaluated in 4-bit NVFP4 precision.

Single Mac Studio: M5 Ultra versus M3 Ultra

On a single Mac Studio with 512GB of unified memory, the M5 Ultra delivers up to 4.6× faster time to first token (TTFT) than the previous-generation M3 Ultra. Apple attributes this to the GPU Neural Accelerator in the M5 generation, which supports dedicated matrix multiplication operations. TTFT is compute bound, so specialized matrix-multiplication hardware translates directly into faster prefill.

Decode speed is memory-bandwidth bound instead. The Mac Studio with M5 Ultra has 1.2 TB/s of unified memory bandwidth, up from 819 GB/s on the M3 Ultra, and Apple reports 1.6× faster token generation as a result.

Qwen 3.8 27B and Mistral Medium 3.5 128B both fit comfortably on a single Mac Studio with 512GB, occupying up to 68GB and 158GB respectively in the large 65,536-token context setup.

Peak power consumption stays in the hundreds of watts

Apple reports peak power consumption during Qwen 3.8 and Mistral Medium 3.5 inference at a 65,536-token context of about 200W for the Mac Studio with M3 Ultra and about 300W for the Mac Studio with M5 Ultra. Apple frames this as "a fraction of the power used by a common toaster" and argues the modest draw means a Mac Studio can be deployed for AI work in any home or office environment.

Why Muse Spark Open cannot fit on one machine

Muse Spark Open would require more than 1.6TB of RAM in its native BF16 precision for the model parameters alone. In 4-bit NVFP4 format the memory footprint drops to about 493GB, but adding the memory needed for intermediate computations during inference still means the model cannot fit on a single machine. It requires parallelization across multiple Mac Studio computers.

Pipeline parallelization: split the model across depth

Pipeline parallelization stores half of the model on one Mac Studio and the other half on another. The first machine executes the first half of the model's layers, then sends the output to the second machine, which executes the second half. Apple describes this scheme as extremely simple to implement, requiring only minimal code changes.

Decode speed under pipeline parallelization is very similar to a single machine, because layers are still evaluated sequentially and only the intermediate outputs between the two layer sets need to be exchanged. The notable exception is very large contexts: at 65,536 tokens, pipeline parallelization is up to about 1.8× faster than a single node for prefill. MLX segments large contexts into smaller chunks — 8,192 tokens in Apple's benchmark, set by --prefill-step-size 8192 — and pipelines prefill across machines, so while the second machine computes the later layers of chunk k, the first machine computes the earlier layers of chunk k + 1.

Tensor parallelization: split the model across width

Tensor parallelization distributes the model's width across several computers. Each machine holds a fraction of every layer, so per-layer computation is parallelized. Apple notes this is more complex to implement than pipeline parallelization and involves many small communication operations: for each input token and each layer, inputs must be broadcast and outputs aggregated. That makes very low-latency communication such as Thunderbolt RDMA, supported by MLX jaccl, critical for the approach to yield any speedup.

Apple reports that dense architectures scale better across multiple computers than MoE models. With a small number of machines, tensor parallelization's communication overhead can offset some of its compute benefits at very large contexts — but for prefill only. This shows up with Muse Spark Open, where separate normalization of the experts' outputs adds prefill communication, enough that at 65,536 tokens two-machine tensor parallelization lags behind two-machine pipeline parallelization. The approach pays off as machines are added: four M5 Ultras deliver the best results. Decode is unaffected, and Muse Spark Open sustains a significant decode speedup under tensor parallelization even at large contexts, comparable to dense models.

Benchmark results as reported by Apple

Time to first token (seconds), lower is better:

Model M3 Ultra M5 Ultra 2 M5 Pipeline 2 M5 Tensor 4 M5 Tensor
Muse Spark Open — — 8.4 8.2 4.8
Mistral Medium 3.5 95.9 24.6 24.5 14.4 7.5
Qwen 3.8 18.2 4.1 4.1 3.1 1.7

Decode speed (tokens per second), higher is better:

Model M3 Ultra M5 Ultra 2 M5 Pipeline 2 M5 Tensor 4 M5 Tensor
Muse Spark Open — — 15.74 24.99 36.73
Mistral Medium 3.5 4.95 7.70 7.70 13.69 23.37
Qwen 3.8 12.46 19.69 19.44 33.27 50.89

Muse Spark Open does not fit on a single machine, so its baseline is the two-M5-Ultra pipeline configuration. Decode speed was measured over 128 generated tokens following the context.

Two Thunderbolt cables per link cut prefill time

In Apple's setup, Mac Studios are connected with two Thunderbolt cables to double link bandwidth. For Muse Spark Open on four M5 Ultras at a 65,536-token context, two cables per link cut TTFT from about 50.5 seconds with one cable to 40.6 seconds — the two-cable value reported in the TTFT table, with the one-cable run being an additional experiment.

How to launch distributed jobs and a local agentic server

MLX provides mlx.launch, a helper command for launching jobs across multiple nodes. A pipeline-parallel benchmark run looks like this:

mlx.launch --hostfile hostfile.json --backend jaccl --env MLX_METAL_FAST_SYNCH=1 -- \
mlx_lm.benchmark --model mistralai/Mistral-Medium-3.5-128B -p 8192 -g 128 -b 1 --prefill-step-size 8192 --pipeline

--backend jaccl selects the RDMA Thunderbolt driver backend. --env MLX_METAL_FAST_SYNCH=1 enables an experimental MLX feature for fast CPU-GPU synchronization that reduces communication overhead between the CPU, which initiates the Thunderbolt communication, and the GPU, which does the compute.

For local agentic coding, mlx_lm.server can be prefixed with mlx.launch to parallelize computation across multiple Mac Studios:

mlx.launch --hostfile hostfile.json --backend jaccl --env MLX_METAL_FAST_SYNCH=1 -- \
mlx_lm.server --model meta-models/Muse-Spark-Open-816B-A42B-NVFP4 --port 8080

Apple shows how to point the OpenCode agentic coding tool at that local server through an OpenAI-compatible provider configuration using baseURL: http://127.0.0.1:8080/v1.

What the Hacker News discussion adds

The Hacker News thread on the post drew 123 comments, and the most useful contributions were skeptical rather than celebratory.

The benchmark comparison drew the sharpest criticism

The most substantive critique concerns the precision mismatch between models. One commenter argued Apple is "sandbagging Qwen pretty heavily by benchmarking Meta's model at 4-bit, against Qwen only at native BF16 (and nothing else shown)." The same commenter added that 4-bit "is not a quant most users would choose" and that Muse Spark Open "isn't even in the top ten models I'd consider if I had a 512 GB M5 in my paws. Or a cluster of them." Another commenter was blunter about the model itself: "A 800B model that performs way worse than other 300B ones. Well..."

Cost and value questions dominated the rest of the thread

Several commenters framed the post as marketing for expensive hardware. One wrote: "So, Meta will spend millions/billions on building open-weight models and then people will send Apple $20,000 to run them. Seems like a good deal for Apple." Another read it as a product launch signal: "Preparing to upsell those very expensive Macs coming later this month, are we." A third noted the cost barrier directly: "It's a good article which showcases direction of travel, i am sure mac studio will get more and more powerful in coming generations but for now it's way to expensive for local setup."

A more optimistic commenter took the long view: "While right now Mac Studio is not a budget device and it certainly costs more than the subsidised subscription, the promise that one day we can run these models on commodity hardware is exciting."

Commenters reported alternative hardware results

One commenter offered independent numbers for other models on Apple silicon: "single M5 Ultra with 256GiB runs DeepSeek 4.1 Flash 0910 just fine (~3200 tok/s prefill, ~45 tok/s). M5 Max w/ 128GiB runs GLM 5.3 Flash also fine." They added that they benchmarked both on quality benchmarks such as MathArena Apex and GPQA Diamond to verify the quality matched official numbers.

Context size and framing complaints

One commenter pointed out that the headline 15–36 tokens per second figures are measured at an 8,192-token context: "Thats not nearly enough context to be useful these days." Another objected to the framing of a four-machine cluster as a desk setup, quoting Apple's own line about running the model "right from your desk" and noting: "Not exactly your average 'desk', but still neat." A third observed that the post discusses Meta's technology without naming Meta outside code references. One commenter also flagged the writing style: "I -- love -- an article -- full of -- em -- dashes."

Key takeaways

  • Apple's MLX distributed inference stack runs Muse Spark Open, an 816B-parameter MoE model with 42B active parameters, at over 36 tokens per second on four Mac Studios with M5 Ultra.
  • The distributed features depend on RDMA over Thunderbolt 5, introduced in macOS 26.2, and MLX v0.30.1 or later, which adds the jaccl communication collectives.
  • Pipeline parallelization splits a model across depth and leaves decode speed essentially unchanged, but speeds up prefill by up to about 1.8× at 65,536-token contexts.
  • Tensor parallelization splits a model across width, scales better for dense architectures, and benefits substantially from two Thunderbolt cables per link.
  • The M5 Ultra's GPU Neural Accelerators deliver up to 4.6× faster TTFT than the M3 Ultra, while its 1.2 TB/s of unified memory bandwidth (versus 819 GB/s) delivers 1.6× faster decode.
  • Peak power consumption is roughly 200W on M3 Ultra and 300W on M5 Ultra during inference at a 65,536-token context.
  • The strongest criticism in the Hacker News thread is that Apple benchmarked Muse Spark Open at 4-bit NVFP4 while benchmarking Qwen 3.8 at native BF16, making the cross-model comparison uneven.

Sources

Related