Swiftlet enables 35B and 80B Qwen models on Mac and iPhone with low RAM usage

Overview

Swiftlet is a Swift + Metal runtime that runs the Qwen3‑Next‑80B‑A3B and Qwen3.6‑35B‑A3B mixture‑of‑experts models on ordinary Apple devices. It keeps only the small dense core of each model resident in memory and streams the routed expert weights from storage on demand, enabling a 35B model to run on an iPhone with about 2.5 GB of RAM and an 80B model to run on a Mac with about 4.3 GB of RAM.

How Expert Streaming Works

The core insight is that each token activates only about 3 B of the model’s parameters. For each layer the model routes the token to a small subset of experts (10 of 512 for the 80B, 8 of 256 for the 35B). Swiftlet:

Keeps the dense weights (attention, DeltaNet projections, routers, shared experts, embeddings) resident: ~1.3 GB for the 35B and ~2.5 GB for the 80B at 4‑bit quantization.
Repacks the tens of thousands of routed experts into fixed‑size expert‑place, and caches hot experts in a bounded pool with LFU plus recency eviction. Cache size has little impact on speed because Apple SSDs absorb misses.
Executes the entire forward pass on Metal using runtime‑compiled shaders, so no Metal toolchain is needed at build time and the same binary runs on iOS.
Uses Gated DeltaNet linear attention for 75 % of the layers, which maintains a fixed‑size recurrent state and avoids a growing KV cache regardless of context length.

Performance and Resource Usage

Model	Disk size	Peak RAM	Decode speed (M5 Mac)	iPhone 17 speed
Qwen3.6‑35B‑A3B (4‑bit)	18 GB	2.6 GB	7‑11 tok/s	~1 tok/s
Qwen3‑Next‑80B‑A3B (4‑bit)	42 GB	4.3 GB	4.5‑5 tok/s	not measured
These numbers come from the project’s README. The low RAM footprint is achieved because only the dense core and a small active expert set are resident; the rest of the weights are streamed from SSD as needed.

Getting Started

To try Swiftlet on a Mac:

Clone the repository and build the release binary:

git clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet
swift build -c release

Download a model container from Hugging Face (resumable):

.build/release/swiftlet-repack \
  --from-hf Leonickson/Qwen3.6-35B-A3B-qpack \
  --output ~/models/qwen3.6-35b.qpack

or for the 80B model:

.build/release/swiftlet-repack \
  --from-hf Leonickson/Qwen3-Next-80B-A3B-qpack \
  --output ~/models/qwen3-next-80b.qpack

Run a chat session:

.build/release/swiftlet chat ~/models/qwen3.6-35b.qpack \
  "Who wrote One Hundred Years of Solitude?" \
  "What language did he write it in?"

Generate text with stats:

.build/release/swiftlet generate ~/models/qwen3.6-35b.qpack \
  --gpu --chat --prompt "Explain expert streaming in one paragraph."

Start an OpenAI‑compatible server (loopback only):

.build/release/swiftlet-server --model ~/models/qwen3.6-35b.qpack --port 8080

The same swiftlet-repack command can repack raw MLX checkpoints (--from-hf mlx-community/... or --source /path/to/checkpoint). Requirements: Apple Silicon, macOS 14+ or iOS 17+, and free SSD space (≈18 GB for 35B, ≈42 GB for 80B).

Usage Options

Swiftlet is designed as a library first, with four main ways to use it:

Swift package – Add SwiftletCore to any macOS or iOS app and use SwiftletSession for chat with streaming deltas, conversation caching, sampling controls, and memory‑pressure handling.
Command‑line interface – swiftlet chat and swiftlet generate for local use and benchmarking; swiftlet-repack to build containers from MLX checkpoints, including streaming straight from Hugging Face with resume capability.
OpenAI‑compatible server – swiftlet-server implements the chat‑completions API on loopback, allowing any UI that talks to an OpenAI‑compatible endpoint to use a streamed local model.
iOS app – The Priv AI app on the App Store embeds SwiftletCore as its streamed‑model engine. Users can download the 35B model via Settings → Experimental Models and chat on‑device with no server involved. The app’s source is available at leonickson1/localLLM; building it yourself requires cloning the Swiftlet repo next to it as swiftlet and opening the Xcode project.

Correctness Validation

Every layer of the forward pass (Gated DeltaNet recurrence, gated GQA attention, sparse MoE routing) is validated against the mlx‑lm reference implementation, both in FP32 and int4 quantized form. Incremental decoding is checked against whole‑sequence processing. Metal kernels are tested against the exact CPU reference, and both fast and scalar GPU kernels produce identical outputs. Containers are byte‑verifiable against their source checkpoints, and streaming placement never changes model semantics: an expert gives the same answer whether served from cache or disk.

Relationship to TurboFieldfare

Swiftlet builds on the expert‑streaming ideas demonstrated by TurboFieldfare for Gemma on Macs. It adopts several published design lessons: streaming experts via pread into a bounded slot pool, evicting with LFU plus recency, packing experts at fixed stride so one fetch equals one read, installing by routing downloaded bytes directly into their final container positions, and compiling shaders at runtime. However, Swiftlet was written from scratch (about 10 k lines of Swift and Metal) and differs in several ways:

Supports the Qwen hybrid stack with Gated DeltaNet linear attention, gated GQA, and high‑sparsity MoE with a shared expert, whereas TurboFieldfare targets the classical dense Gemma architecture.
Implements MLX affine int4/int8 group quantization in Metal, using byte‑addressed kernels with 64‑bit offsets for multi‑gigabyte shards, a cooperative simdgroup GEMV fast path, and explicit hazard management.
Provides a validated CPU reference implementation and fixture infrastructure that guards every kernel change.
Includes the .qpack container and repacker, a resumable Hugging Face streaming installer with stall recovery and download cancellation.
Adds a chat session layer that handles thinking and non‑thinking Qwen variants, sampling with presence and frequency penalties, minimum‑length and sentence‑completion stopping, conversation caching with delta prefill, and iOS memory‑pressure coordination.
Delivers end‑to‑end iPhone support, including app‑engine integration. The project acknowledges inspiration from colibrì for caching and placement policy, and uses mlx‑lm as the correctness reference throughout. Swiftlet was built in collaboration with Claude Code.

Community Feedback and Limitations

Comments on the Hacker News post highlight both enthusiasm and concerns:

Optimistic view: some see this as progress toward a future where consumer devices can run large models efficiently, with one commenter noting "this is how progress happens" and another stating "Running a 35B on iPhone at 1 tok/s with 2.5 GB RAM… this is the future of on-device inference."
Concerns about storage wear and speed: several users warn that frequent expert streaming could wear out SSDs and that decode speeds are low (e.g., "At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell." and "prefill becomes the bottleneck… half an hour to process 10k tokens on an M5 seems… not great" ).
Desire for tunable RAM usage: a user with a 32 GB Mac asked whether RAM usage could be made configurable to exploit extra memory for faster execution.
Questions about the Claude Code collaboration: commenters wondered whether the claim "built in collaboration with Claude Code" indicates a formal partnership with Anthropic or merely a note about AI‑assisted development.
Comparisons to other projects: links to similar streaming‑weight efforts (e.g., a 400B model claim on iPhone) and references to alternative implementations such as BigMoeOnEdge were shared.
Platform‑specific inquiries: users asked about Android/Linux/Windows support and whether the approach could be ported elsewhere.

These remarks reflect the trade‑off inherent in expert‑streaming: low RAM footprint at the cost of increased SSD traffic and latency, especially during the prefill phase.

Conclusion

Swiftlet demonstrates that Mixture‑of‑Experts models can be run on consumer Apple devices by keeping only a small dense core in memory and streaming the routed experts from storage. The approach enables a 35B Qwen model to operate on an iPhone with ~2.5 GB RAM and an 80B model on a Mac with ~4.3 GB RAM, achieving token‑per‑second rates suitable for experimental on‑device chat. While the method reduces RAM requirements, it shifts the bottleneck to storage bandwidth and wear, leaving room for future work on caching strategies, prefetch optimizations, and hardware‑accelerated expert access.