SharpAI/SwiftLM

What it is

SwiftLM is a high-performance inference server written in Swift that runs MLX (Apple's machine learning framework) models on Apple Silicon. It presents a standard OpenAI-compatible API (/v1/chat/completions), so existing clients and tools can talk to it without modification. Unlike the typical Python-based stacks, it is compiled to a single native binary with no Python runtime and no GIL; the README emphasizes bare-metal performance on Apple Silicon.

It is a serious, actively-developed project with deep engineering details: benchmark tables, bug-fix explanations (memory bugs exposed on 32 GB Macs), and performance features like MTP speculative decoding bit-identical rollback after a sliding-window boundary bug, and --stream-experts SSD streaming for oversized MoE models.

Key capabilities & innovations

1. OpenAI-compatible API — drop-in replacement for OpenAI-style endpoints; models download automatically from Hugging Face (e.g., mlx-community/gemma-4-26b-a4b-it-4bit).

2. MoE (Mixture-of-Experts) optimization — For models like Gemma 4 26B (which only activates ~4B of 26B parameters per token), SwiftLM reads only the active experts, achieving 5–6× higher decode speed than a dense 27B model at the same memory footprint.

3. TurboQuant — The headline feature: compresses the KV cache 4×. At 100K context on Gemma 4-26B 4-bit, it delivers 66.9 tok/s vs 27.5 tok/s baseline (2.43× speedup) and cuts OS RAM at 40K context from 48.7 GB to 18.2 GB. At 8-bit, it gives 48.3 tok/s vs 14.9 vanilla (3.24×). The README explicitly shows that TurboQuant is often more effective than MTP on bandwidth-bound models.

4. MTP (Multi-Token Prediction / speculative decoding) — Predicts multiple tokens per stepenter validation. On 8-bit models (bandwidth-bound), MTP gives +20% at 40K context and +51% at 100K. The README carefully notes when MTP doesn't help (4-bit MoE is compute-bound; combine with TurboQuant it loses because compute is the overhead).

5. SSD streaming (--stream-experts) — For models too large for unified memory, SwiftLM can stream expert layers directly from NVMe SSD. E.g., a 126 GB DeepSeek-V4-Flash runs on a 64 GB M5 Pro at ~4.2 tok/s at 40K context with TurboQuant, vs 0.32 tok/s without.

6. Long-context support — Tested up to ~80.7K tokens on a base Mac mini M6 (32 GB) and 100K on M5 Pro, with careful attention to KV cache sizing and the --turbo-kv long-range recall fix (#175, #65).

7. Memory guards & correctness — The README documents how 32 GB machines exposed hidden bugs (unbounded MLX buffer cache, wrong KV-cache estimates that crashed with Metal GPU timeout, VLM fallback when preprocessor_config.json lacks image_mean, chunked prefill fixes). It also notes DFlash (a different speculative-decoding project) is unsuitable for production on Qwen3.6 due to greedy decoding lock-in, and points to the proper fix (stochastic posterior sampling per Leviathan/Chen).

Why it matters

• Native Apple Silicon performance — No Python overhead; a single binary uses Metal, unified memory, and the full bandwidth of M-series chips.
• Memory efficiency — Makes large MoE models (e.g., 26B) runnable on 32 GB Macs, and even 126 GB models on 64 GB via SSD streaming.
• Speedups at long context — TurboQuant and MTP are combined in careful, benchmark-driven ways; the README includes tables showing when to use which (--mtp for 8-bit at 40K+, --turbo-kv for maximum throughput with TurboQuant, --stream-experts for 24 GB Macs).

Practical use cases

• Running LLMs locally on Mac for development, testing, or production with an OpenAI-compatible endpoint.
• Deploying on laptops/desktops where GPU memory is limited (e.g., a 24 GB Mac using --stream-experts).
• Long-context applications (code files, documents) that need 40K–100K tokens without exhausting RAM.
• iOS integration (there's a SwiftBuddy iOS demo).

Technical notes (to be fair)

• The README is dense with benchmark tables across different chips (M1 Ultra, M5 Pro, M6) and model types (4-bit vs 8-bit, MoE vs dense), with clear caveats: "TTFT excluded", "memory guard aborts swaps >2 GB", "needle-check unreliable at 40.8K in both modes".
• It's honest about limitations: --gpu-layers on MoE is very slow (single-core CPU fallback), MTP+TurboQuant can undercut TurboQuant alone on compute-bound models, and DFlash is flagged as non-production.
• Build requires Homebrew's cmake, compiles Metal kernels from submodules (mlx-swift-lm fork), and the release tarball is self-contained.

In short: a serious, well-engineered, benchmark-driven Swift server for Apple Silicon LLM inference, with the standout TurboQuant KV compression and careful MoE/SSD streaming support.