slotstream: Running 104GB Qwen3.8-Flash-Next on 48GB Macs

slotstream allows Apple Silicon Mac users to run the 104GB Qwen3.8-Flash-Next (125B MoE, 4-bit) model on hardware with significantly less RAM than the model's total size. By streaming routed experts from the SSD to a managed memory pool, the tool enables inference on machines with as little as 8.1 GB of available memory, though performance scales with available RAM.

Performance and Hardware Requirements

On a 48 GB Mac (M5 Pro), slotstream achieves a warm decode speed of approximately 12 tokens per second with a cold start time to the first token of about 3 seconds. The system is designed to be elastic, automatically sizing its memory target to the lowest of 33 GB, 70% of total RAM, or the Metal working-set limit.

Estimated Performance by RAM Tier

Memory	Estimated Speed
16 GB	~5 tok/s
24 GB	~8 tok/s
32 GB	~10 tok/s
48 GB+	~12 tok/s

Disk space is the primary constraint. Users require approximately 110 GB of free disk space to store the model weights. Because the weights are streamed from the SSD, the speed of the storage medium directly impacts performance; users running the model from external USB disks have reported speeds as low as 0.5 tokens per second.

Technical Architecture: SSD Streaming

slotstream avoids the memory exhaustion that typically occurs when loading large MoE (Mixture-of-Experts) models. While standard mlx_lm.load() attempts to load the entire model into memory, slotstream utilizes a custom streaming approach:

• Resident Components: The dense trunk (3.8 GB) and a 32 GB n-gram table are kept resident in memory.
• Expert Streaming: The 68 GB of routed experts (512 per layer) are read using pread into a fixed pool of cache slots shared across all 48 layers.
• Memory Elasticity: The system re-checks available memory every 15 seconds, resizing the expert cache between requests to shrink under pressure and grow when resources are available.

This architecture ensures that greedy decoding remains byte-identical regardless of the cache size; changing the amount of allocated RAM affects only the speed of inference, not the output quality.

Installation and API Compatibility

slotstream is distributed as a single Swift binary. It requires macOS 14+ and Apple Silicon.

Deployment

Users can install the tool via a curl script that places the binary in ~/.slotstream/bin. The tool includes a doctor command to preview the memory plan and disk availability before downloading the 104 GB of weights.

API and Integration

The serve command implements a subset of the Ollama and OpenAI chat/generate endpoints on port 11434. This allows slotstream to be used with existing Ollama clients, the OpenAI SDK, and Open WebUI. Supported sampling options include temperature, top_p, top_k, min_p, presence_penalty, seed, num_predict, and stop.

Current Limitations

• Prefill Latency: Long prompts are slow to start because the entire prompt must be processed before the first token is generated. Prefill speeds range from ~50 tok/s (16 GB Mac) to ~125 tok/s (48 GB Mac). However, follow-up turns in a conversation reuse the previous state, keeping the time to first token flat as the chat grows.
• Hardware Validation: While derived from a 48 GB M5 Pro, the performance estimates for smaller RAM tiers are based on a measured curve rather than real-world testing on 16 GB or 24 GB hardware.
• OS Compatibility: The installer has been tested on macOS 14 and 15, but the runtime has not been fully exercised on these versions.

Community Insights and Counterpoints

Community discussion highlights a tension between the desire for democratized local LLM access and the redundancy of fragmented implementations.

"There are already a handful of repos doing essentially exactly this... I'd much rather see people collaborate on one of these implementations... instead of producing yet another near-identical repo."

Other users have noted that while 12 tokens per second is usable, the high disk I/O required for streaming experts may raise concerns regarding SSD wear over time, and that the efficiency of the 8.1 GB memory floor is a critical metric for users with entry-level hardware.