SharpAI/SwiftLM

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.

What it solves

SwiftLM is a high-performance inference server designed specifically for Apple Silicon. It eliminates the overhead of Python runtimes and the Global Interpreter Lock (GIL), allowing users to run large language models (LLMs), vision-language models (VLMs), and audio-language models (ALMs) with bare-metal performance on macOS and iOS.

How it works

Built natively in Swift and Metal, SwiftLM serves MLX models using an OpenAI-compatible API. It employs several advanced optimization techniques to handle massive models on limited hardware:

  • SSD Expert Streaming: For Mixture of Experts (MoE) models, it streams expert weights directly from the NVMe SSD to the GPU, enabling models like Qwen3.5-397B to run on machines with as little as 64GB of RAM.
  • TurboQuantization: A hybrid KV cache compression architecture that reduces memory usage by ~3.5x compared to FP16 with minimal accuracy loss, using non-linear Lloyd-Max codebooks processed in fused Metal shaders.
  • Speculative Decoding: Uses a small draft model to predict tokens that are then verified in bulk by a larger main model, accelerating in-RAM inference.
  • Native Multimodal Support: Includes dedicated flags for vision (--vision) and audio (--audio) processing, supporting real-time image parsing and audio ingestion via AVFoundation.

Who it’s for

  • Apple Silicon Users: Developers and researchers wanting the fastest possible local LLM inference on Mac or iPhone/iPad.
  • Memory-Constrained Users: Those needing to run frontier-scale MoE models (100B+ parameters) on consumer-grade Apple hardware.
  • iOS Developers: Those looking to integrate on-device MLX inference into mobile applications via the included SwiftBuddy companion app.

Highlights

  • OpenAI-Compatible: Drop-in replacement for OpenAI SDKs.
  • 100% Native: No Python required; compiled to a single binary.
  • Massive MoE Support: SSD streaming provides up to a 10x speedup for oversized models.
  • KV Cache Compression: TurboQuant allows processing 100k context windows on 24GB RAM machines.
  • Multimodal: Native support for VLMs (e.g., Qwen2-VL) and ALMs (e.g., Gemma 4 Omni).