carloslfu/slotstream

Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16–64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.

What it solves

Slotstream lets you run a very large AI model (Qwen3.8-Flash-Next, 125 billion parameters) on Macs that normally could not hold it in memory. Most Macs have 16–64 GB of RAM, but this model would need far more to fit entirely in memory. Slotstream keeps most of the model on the SSD and loads only the parts needed at each moment, so you can chat, ask about images, and run coding agents locally and offline.

How it works

The model is a mixture-of-experts: generating each piece of text uses only a subset of its expert networks. Slotstream keeps shared weights in RAM and reads the needed experts from SSD into a cache. Frequently used experts stay in memory, reducing repeated disk reads. It automatically picks a memory budget and context window for your Mac, and a governor resizes the cache as other apps need room. The whole engine is one native Swift program built on Apple's MLX and Metal, with no Python runtime between request and GPU.

Who it's for

People with Apple Silicon Macs (16 GB or more, macOS 14+) who want to run a frontier-class open model without a cloud account or a huge memory machine. It also suits developers who want to connect their own apps via OpenAI-, Ollama-, or Anthropic-compatible APIs, or use it as a backend for coding agents like Claude Code, Codex, Pi, opencode, and Hermes. It is not optimized for 96 GB+ Macs where the model fits in memory.

Highlights

  • Runs a 125B-parameter model on 16–64 GB Macs by streaming experts from SSD
  • Measured 20.91–23.69 tokens per second on a 48 GB M5 Pro at a 33 GB memory budget
  • Works fully offline after a one-time model download
  • Native Swift on MLX and Metal — no Python, one-command install
  • Supports images, coding agents, and OpenAI/Ollama/Anthropic-compatible APIs
  • Automatic memory planning and cache resizing as other apps need RAM

Related

  • Dispatch
    slotstream: Running 104GB Qwen3.8-Flash-Next on 48GB Macs

    slotstream is a Swift-based tool that enables Apple Silicon Macs to run the 104GB Qwen3.8-Flash-Next model by streaming experts from SSD to RAM, achieving up to 12 tokens per second on 48GB hardware.

  • Project
    SharpAI/SwiftLM

    A native Swift, OpenAI-compatible inference server for MLX models on Apple Silicon, featuring TurboQuant (4× KV-cache compression), MTP speculative decoding, and SSD streaming for large MoE models, with extensive benchmarks on M-series chips.

  • Dispatch
    Swiftlet enables 35B and 80B Qwen models on Mac and iPhone with low RAM usage

    Swiftlet lets you run the 35B Qwen model on an iPhone using ~2.5 GB RAM and the 80B Qwen model on a Mac using ~4.3 GB RAM by streaming Mixture‑of‑Experts weights from storage.

  • Project
    leonickson1/Swiftlet

    A Swift and Metal runtime that enables large MoE models like Qwen to run on Apple devices by streaming experts from storage to RAM on demand.

  • Project
    Niko1221/Strata

    Strata is a free, MIT-licensed app that runs AI models locally on your own computer — no account or cloud needed. It offers a desktop app and a terminal app, supports several model families (Qwen and DeepSeek variants) with documented hardware requirements and speed benchmarks, and connects to coding tools like Claude Code and Codex CLI through OpenAI/Anthropic-compatible APIs. Works on NVIDIA, AMD, Intel Arc, Apple Silicon, and Ryzen AI Max hardware.