Swiftlet: Running 80B and 35B Qwen Models on Mac and iPhone

Swiftlet enables the execution of large-scale Mixture-of-Experts (MoE) models, specifically the Qwen3-Next and Qwen3.5/3.6 families, on consumer Apple hardware. By streaming routed weights from storage on demand rather than loading the entire model into memory, Swiftlet allows an 80B parameter model to run in 4.3 GB of RAM on a Mac and a 35B parameter model to run in 2.5 GB of RAM on an iPhone.

Expert Streaming Architecture

Swiftlet achieves its low memory footprint by exploiting the sparsity of MoE models, where only a small fraction of parameters are active for any given token. In the Qwen hybrid models, only about 3B parameters are active per token.

Memory Management and Weight Streaming

To minimize RAM usage, Swiftlet employs a specific memory strategy:

  • Resident Dense Weights: The runtime keeps essential dense weights—including attention, DeltaNet projections, routers, shared experts, and embeddings—permanently in memory. This requires approximately 1.3 GB for the 35B model and 2.5 GB for the 80B model at 4-bit quantization.
  • Fixed-Stride Blobs: Routed experts are repacked into a .qpack container using fixed-stride blobs. This allows the system to fetch a specific expert using a single pread operation from the SSD, avoiding the overhead of mmap and page-cache thrashing.
  • LFU Caching: A bounded pool caches "hot" experts using a Least Frequently Used (LFU) policy combined with recency eviction. Because Apple SSDs provide high throughput, the system maintains performance even with cache hit rates ranging from 43% to 70%.

Hardware Acceleration

Swiftlet is written in Swift and Metal, utilizing runtime-compiled shaders to execute the forward pass on the GPU. This approach removes the need for a Metal toolchain at build time and ensures compatibility across both macOS and iOS.

Performance and Model Support

Swiftlet supports 4-bit quantized versions of the Qwen MoE hybrid family. The performance varies by device and model size:

Model Disk Space Peak RAM Decode Speed (M5 Mac)
Qwen3.6-35B-A3B 18 GB 2.6 GB 7 to 11 tok/s
Qwen3-Next-80B-A3B 42 GB 4.3 GB 4.5 to 5 tok/s

On an iPhone 17, the 35B model runs in approximately 2.5 GB of RAM at a speed of roughly 1 token per second.

Architectural Trade-offs

While the models maintain the chat and writing capabilities of large models, the author notes that because only 3B parameters are active per token, they "recall facts like small ones."\n

Implementation and Tooling

Swiftlet is designed as a library first, providing four primary ways to interact with the models:

  1. Swift Package: SwiftletCore can be integrated into macOS or iOS apps via SwiftletSession for chat, streaming deltas, and memory-pressure handling.
  2. CLI Tools: swiftlet chat and swiftlet generate for local use, and swiftlet-repack for converting MLX checkpoints into .qpack containers.
  3. OpenAI-Compatible Server: swiftlet-server provides a loopback API, allowing any OpenAI-compatible UI to connect to the local model.
  4. iOS App: The Priv AI app on the App Store embeds SwiftletCore to provide a native on-device chat experience.

Technical Lineage and Correctness

Swiftlet builds upon the "expert-streaming thesis" proven by TurboFieldfare for Gemma models. It adopts several design lessons from TurboFieldfare, including the use of pread for streaming, LFU eviction, and runtime shader compilation.

To ensure correctness, every layer of the forward pass—including Gated DeltaNet recurrence and sparse MoE routing—is validated against mlx-lm reference implementations using per-layer fixtures in both f32 and int4 quantized forms.

Community Perspectives

Discussion among users highlights both the potential and the practical limitations of this approach:

  • Hardware Wear: Some users expressed concern that constant disk swapping could lead to premature SSD wear ("NAND burners").
  • Prefill Bottlenecks: Critics noted that while decode speeds are acceptable, the prefill stage (processing the initial prompt) may become a significant bottleneck for long contexts.
  • Scalability: Some users suggested that the RAM cache should be tunable, allowing users with more RAM (e.g., 32 GB) to increase the cache size and further improve speed.
  • Future of On-Device AI: Proponents argue that this is a necessary step toward a future where massive models run on consumer hardware, potentially replacing expensive GPU clusters with affordable SSD-based storage.

Sources