SharpAI/SwiftLM

What it solves

SwiftLM is a high-performance inference server designed to run large language models (LLMs) on Apple Silicon without the overhead of a Python runtime or the Global Interpreter Lock (GIL). It enables the execution of massive models—including those that exceed physical RAM—by utilizing native Swift and Metal, providing an OpenAI-compatible API for easy integration.

How it works

The project leverages the MLX framework and compiles to a single native binary. It employs several advanced optimization techniques to maximize Apple Silicon performance:

• TurboQuantization: A hybrid KV cache compression architecture that reduces memory usage by ~3.5x using non-linear Lloyd-Max codebooks and fused Metal shaders.
• SSD Expert Streaming: A specialized pipeline for Mixture of Experts (MoE) models that streams expert layers directly from NVMe SSD to GPU, allowing models like Qwen3.5-397B to run on limited RAM.
• Speculative Decoding: Uses a small draft model to predict tokens that are verified in bulk by a larger main model to increase throughput.
• Native Multimodal Support: Includes dedicated flags for vision (--vision) and audio (--audio) processing using Metal and AVFoundation.

Who it’s for

• Apple Silicon users wanting to run frontier open-weights models locally with maximum efficiency.
• Developers needing a drop-in OpenAI-compatible local server for their applications.
• iOS Developers looking to implement on-device MLX inference (via the included SwiftBuddy companion app).

Highlights

• 100% Native: No Python required; compiled to a single binary for macOS and iOS.
• OpenAI-Compatible: Supports /v1/chat/completions and streaming.
• Massive Model Support: Capable of running 100B+ parameter MoE models via SSD streaming.
• Broad Model Compatibility: Supports a wide array of families including Gemma 4, Qwen 3.5, Llama 3.3, DeepSeek V3, and Phi 4.
• KV Cache Compression: Significant memory savings at long contexts via TurboQuant.