carloslfu/slotstream

slotstream – Run a 125‑billion‑parameter LLM on a Mac with 48 GB RAM

What it is – A native Swift command‑line tool (and library) that lets you run the Qwen‑3.8‑Flash‑Next 4‑bit model (≈105 GB on disk, 125 B parameters) on Apple‑silicon Macs. It streams most of the model’s weights from an SSD into a small RAM cache, so a 48 GB Mac can generate ~12 tokens / s after the cache warms up. The binary ships with no Python, works offline after the one‑time download, and exposes Ollama‑compatible and OpenAI‑compatible HTTP APIs.

Why it matters – Large‑scale LLMs usually need dozens of gigabytes of GPU memory. slotstream shows that, with clever expert‑caching and 4‑bit quantisation, a consumer‑grade Mac can run a 125 B‑parameter model without swapping the whole model into RAM.

---

Key features (as described in the README)

• Model loading from SSD – Weights are stored in a compact 4‑bit format (≈88 GB compressed). slotstream keeps a fixed pool of expert slots in RAM and pulls the needed pieces from the SSD on demand.
• Automatic memory planning – On start‑up the program picks a memory target based on your Mac’s RAM, the Metal working‑set limit, and a 70 %‑of‑RAM ceiling. You can override it with --memory‑gb.
• Fast inference – On a 48 GB M5 Pro Mac you get ~12 tok/s after the cache is warm; smaller machines still work (≈3 tok/s on 8 GB RAM, though they may swap).
• API compatibility – Provides:
  • An Ollama‑style endpoint (/api/chat) that works with Open WebUI and the Ollama CLI.
  • An OpenAI‑style endpoint (/v1/chat/completions) usable from any OpenAI SDK.
  • Image support (base‑64 or data‑URL payloads) with a vision tower that adds ~0.9 GB to the memory plan.
  • Tool‑calling via the OpenAI chat schema and via the Vercel AI SDK gateway used by the fx agent framework.
• CLI commands
  • slotstream pull – download and verify the model from a public Hugging Face mirror (resume‑able, hash‑checked).
  • slotstream run --prompt "…" – one‑off generation from the terminal.
  • slotstream serve – start the HTTP server for client integration.
  • slotstream doctor – show the memory plan, estimated speed, and free‑disk check before downloading.
• Swift package – The same engine can be added to a Swift project via Package.swift, exposing Planner and WeightStore APIs for custom apps.
• Speculative decoding – An optional 1.5 GB “draft head” can be loaded to propose tokens that the main model verifies, giving up to ~24 % speed‑up when enough RAM is available.
• Extensive documentation – Guides for installation, client setup, vision, coding agents (fx), Hermes tool‑calling, troubleshooting, hardware measurements, and design details.

---

System requirements

Requirement	Details
CPU / GPU	Apple Silicon (M‑series) with Metal support. macOS 14+ (tested on 14, 15, 26).
RAM	Minimum 8 GB (will swap heavily). Recommended 48 GB for full speed.
Disk	~110 GB free SSD space for the compressed model (≈88 GB) plus the binary.
Other	No Python needed; the binary is a self‑contained Swift executable.

---

Typical workflow

1. Install – curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh (places slotstream in ~/.slotstream/bin).
2. Check your Mac – slotstream doctor to see the auto‑chosen memory target and free‑space.
3. Download the model – slotstream pull (one‑time, resumable, hash‑verified).
4. Run a quick test – slotstream run --prompt "Why is the sky blue?".
5. Serve for apps – slotstream serve and point Open WebUI, Ollama, or any OpenAI SDK at http://localhost:11434.
6. Optional – Enable vision (--vision on), speculative decode (--mtp auto), or adjust context length (--max-context 65536).

---

Limitations & caveats

• Model‑only support – Currently only qwen3.8‑flash‑next:4bit is implemented; other LLM families are not compatible.
• Hardware bound – Requires Apple‑silicon and Metal; no Linux/Windows support yet (planned for a future “Sevra” product).
• Memory pressure – On low‑RAM Macs the system may swap heavily, slowing generation and potentially wearing out the SSD.
• Context window – Default 32 k tokens (can be raised to 65 k with a flag). Long prompts still incur a multi‑second pre‑fill cost.
• Vision accuracy – Image handling works, but the project does not provide a benchmark for visual QA.
• Concurrency – Only one generation at a time per process; multiple users would need separate processes.

---

Who might use it

• Developers building local‑first AI assistants (e.g., the upcoming Sevra app) who want a fast, offline LLM on a Mac.
• Researchers experimenting with Mixture‑of‑Experts models on consumer hardware.
• Power users who prefer a self‑contained binary and want to avoid Python/conda environments.
• Tool‑building enthusiasts integrating with Ollama, OpenAI SDKs, or the fx agent framework.

---

Quick reference links

• Installation script – install.sh
• API docs – docs/API.md
• Client guide – docs/CLIENTS.md
• Vision guide – docs/API.md#images
• Coding‑agent guide – docs/FX.md
• Design & memory planner – PLAN.md
• Performance measurements – MEASUREMENTS.md
• Release notes – CHANGELOG.md

---

Bottom line

slotstream is a real, production‑grade inference engine that makes a 125 B‑parameter LLM usable on a single Apple‑silicon Mac by streaming weights from SSD and managing a dynamic cache. It ships as a tiny Swift binary, offers standard Ollama/OpenAI endpoints, supports images and tool‑calling, and provides a Swift library for embedding the engine in native macOS apps. The project is actively maintained, with detailed documentation and a clear hardware‑focused performance model.