Swiftlet: Running 80B and 35B Qwen MoE Models on Mac and iPhone

Swiftlet enables the execution of large Mixture-of-Experts (MoE) models, specifically the Qwen3-Next and Qwen3.5/3.6 families, on consumer Apple hardware by streaming model weights from disk. This approach allows an 80B parameter model to run using only 4.3 GB of RAM on a Mac and a 35B parameter model to run on an iPhone using approximately 2.5 GB of RAM.

Expert Streaming Architecture

Swiftlet achieves low memory footprints by keeping only the dense core of the model resident in memory while streaming routed experts from storage on demand.

Memory Management and Storage

Swiftlet utilizes a .qpack container format that repacks routed experts into fixed-stride blobs. This design ensures that fetching a single expert requires exactly one pread operation from the SSD, avoiding the overhead of mmap and page-cache thrashing.

To optimize performance, Swiftlet employs a bounded pool for caching "hot" experts, using a Least Frequently Used (LFU) policy combined with recency eviction. According to the project documentation, Apple SSDs are efficient enough to absorb misses, with hit rates between 43% and 70% maintaining consistent throughput.

Hardware Acceleration

The runtime is built using Swift and Metal, employing runtime-compiled shaders to ensure the same code operates across both macOS and iOS without requiring a Metal toolchain at build time. The forward pass is executed entirely on the GPU via Metal.

Performance and Model Specifications

Swiftlet supports the Qwen hybrid stack, which combines Gated DeltaNet linear attention, gated GQA, and high-sparsity MoE with a shared expert. Because 75% of the layers use Gated DeltaNet linear attention with a fixed-size recurrent state, the system avoids a growing KV cache regardless of context length.

Benchmarks

Model Disk Space Peak RAM Decode Speed (M5 Mac)
Qwen3.6-35B-A3B (4-bit) 18 GB 2.6 GB 7 to 11 tok/s
Qwen3-Next-80B-A3B (4-bit) 42 GB 4.3 GB 4.5 to 5 tok/s

On an iPhone 17, the 35B model runs natively using approximately 2.5 GB of RAM at a speed of roughly 1 token per second.

Functional Trade-offs

While these models behave like large models in terms of chatting and writing, the author notes a specific limitation regarding knowledge: because only about 3B parameters are active per token, the models recall facts like smaller models.

Deployment and Integration

Swiftlet is designed as a library first, providing four primary ways to integrate and use the model:

  1. Swift Package: The SwiftletCore package can be added to macOS or iOS apps for chat functionality with streaming deltas and memory-pressure handling.
  2. Command Line Interface (CLI): Tools include swiftlet chat and swiftlet generate for local use, and swiftlet-repack for converting MLX checkpoints into .qpack containers.
  3. OpenAI-Compatible Server: swiftlet-server provides a loopback API that allows any OpenAI-compatible UI to interface with the local streamed model.
  4. iOS Application: The Priv AI app on the App Store embeds SwiftletCore as its engine for on-device, serverless chat.

Technical Lineage and Correctness

Swiftlet adopts several design lessons from TurboFieldfare, specifically the use of pread for expert streaming into a bounded slot pool and the use of fixed-stride packing for experts. However, Swiftlet is a ground-up implementation (approximately 10,000 lines of Swift and Metal) that supports the Qwen hybrid architecture, whereas TurboFieldfare focuses on Gemma (a dense transformer).

To ensure correctness, every layer of the forward pass—including Gated DeltaNet recurrence, gated GQA attention, and sparse MoE routing—is validated against mlx-lm reference implementations using per-layer fixtures in both f32 and int4 quantized forms.

Community Discussion and Perspectives

Community feedback on the project highlights a tension between technical achievement and practical utility:

  • Hardware Longevity: Some users expressed concern that frequent disk swapping (streaming weights from SSD) could lead to premature wear of the NAND flash storage.
  • Prefill Bottlenecks: Critics noted that while decode speeds are acceptable, the prefill stage (processing the initial prompt) may become a significant bottleneck for long contexts.
  • Potential for Scaling: Other users suggested that the system could be further optimized by making the RAM cache tunable, allowing users with higher-RAM Macs (e.g., 32GB) to increase the cache size to improve speed.

"I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips."

"Running a 35B on iPhone at 1 tok/s with 2.5 GB RAM… this is the future of on-device inference."

Sources