leonickson1/Swiftlet

Swiftlet is a Swift and Metal runtime that runs large Qwen Mixture-of-Experts models locally on Apple devices by streaming expert weights from storage, enabling 35B and 80B models to run with low RAM, including on iPhone.

What it solves

Swiftlet allows large Mixture-of-Experts (MoE) models, such as the Qwen3-Next and Qwen3.5/3.6 families (up to 80B or even 397B parameters), to run on consumer Apple hardware, including iPhones. It solves the memory bottleneck by streaming model weights from the SSD to the GPU on demand rather than requiring the entire model to be resident in RAM.

How it works

Swiftlet uses a hybrid approach to memory management:

  • Dense Core Residency: It keeps only the essential dense weights (attention, embeddings, routers) resident in memory.
  • Expert Streaming: Routed experts are packed into a .qpack container with fixed strides. When a token is processed, the runtime uses pread to fetch only the required experts from storage into a bounded cache pool.
  • Metal Integration: The entire forward pass is executed on the GPU using runtime-compiled Metal shaders.
  • Linear Attention: It leverages Gated DeltaNet linear attention in 75% of the layers, which prevents the KV cache from growing as context length increases.

Who it’s for

  • Apple Silicon Users: Developers and enthusiasts with Macs (M1 through M5) or iPhones (iOS 17+) who want to run high-parameter models locally.
  • iOS App Developers: Those looking to integrate a high-performance local LLM engine into Swift-based applications via the SwiftletCore package.

Highlights

  • Extreme RAM Efficiency: Runs an 80B model in ~4.3 GB of RAM and a 35B model in ~2.6 GB of RAM (4-bit).
  • Cross-Platform: Works across macOS 14+ and iOS 17+.
  • Multiple Interfaces: Provides a Swift package for app integration, a CLI for benchmarking, and an OpenAI-compatible server for third-party UIs.
  • Validated Correctness: Every layer is validated against mlx-lm reference implementations to ensure output parity.
  • iPhone Support: Enables the execution of 35B parameter models on a base iPhone.

Related

  • Dispatch
    Swiftlet enables 35B and 80B Qwen models on Mac and iPhone with low RAM usage

    Swiftlet lets you run the 35B Qwen model on an iPhone using ~2.5 GB RAM and the 80B Qwen model on a Mac using ~4.3 GB RAM by streaming Mixture‑of‑Experts weights from storage.

  • Project
    SharpAI/SwiftLM

    A native Swift inference server for Apple Silicon that serves MLX models via an OpenAI-compatible API, featuring SSD expert streaming and KV cache compression.

  • Project
    drumih/turbo-fieldfare

    TurboFieldfare is a Swift + Metal runtime that streams the Gemma 4 26B‑A4B model from SSD, enabling inference on Apple‑silicon Macs with as little as 8 GB RAM. It ships a native macOS app, a command‑line tool, a Swift library, and a loop‑back OpenAI‑compatible server, all under Apache 2.0.

  • Dispatch
    slotstream: Running 104GB Qwen3.8-Flash-Next on 48GB Macs

    slotstream is a Swift-based tool that enables Apple Silicon Macs to run the 104GB Qwen3.8-Flash-Next model by streaming experts from SSD to RAM, achieving up to 12 tokens per second on 48GB hardware.

  • Project
    carloslfu/slotstream

    slotstream is a Swift‑based inference engine that runs the 125‑billion‑parameter Qwen‑3.8‑Flash‑Next model on Apple‑silicon Macs. It streams 4‑bit‑quantised weights from SSD into a RAM cache, enabling ~12 tokens / s on a 48 GB Mac. The tool offers a CLI, Ollama‑compatible and OpenAI‑compatible HTTP APIs, optional vision support, speculative decoding, and a Swift package for embedding in apps.