TurboFieldfare: Running Gemma 4 26B on M-Series Macs with 2 GB RAM
TurboFieldfare enables the execution of the Gemma 4 26B-A4B model on any Apple Silicon Mac, including base models with only 8 GB of RAM, by reducing the active memory footprint to approximately 2 GB. It achieves this by keeping only the shared core and KV cache in memory while streaming the required MoE (Mixture of Experts) weights from the SSD in real-time.
SSD-Backed Expert Streaming Architecture
TurboFieldfare bypasses the need to load the entire 14.3 GB model into RAM by implementing a custom streaming mechanism for routed experts.
Memory Management and I/O
At each transformer layer, the engine computes attention and the router using resident weights. The CPU uses the router's top-8 expert IDs to plan against a 16-slot LFU (Least Frequently Used) cache. If the required experts are not in the cache, the engine performs bounded parallel pread calls to load them into Metal-visible buffers. To maximize efficiency, Metal computes the resident shared-expert branch concurrently while the SSD reads are in progress.
Weight Quantization and Layout
To further minimize the memory footprint, the engine uses specific quantization formats:
- Weights: MLX affine 4-bit (group 64) for embeddings, attention, shared-experts, and routed-experts.
- Router: 8-bit quantization.
- KV Cache: FP16 storage with a sliding-window approach for 25 layers and linear storage for 5 full-attention layers.
Performance Benchmarks
Throughput varies significantly based on the hardware generation and available system RAM, which affects the OS page cache's ability to keep experts resident.
| Hardware | Measured Decode Speed |
|---|---|
| M2 MacBook Air (8 GB RAM) | 5.1–6.3 tok/s |
| M4 Mac mini (16 GB RAM) | ≅5 tok/s |
| M5 Pro (24 GB RAM) | 31–35 tok/s |
| M4 Max (64 GB RAM) | ≅48 tok/s |
Users with higher RAM (e.g., 64 GB) report significantly higher speeds because the OS page cache can keep the entire 12 GB packed_experts set resident in memory, effectively eliminating SSD latency.
Implementation and Tooling
TurboFieldfare is a model-specific runtime written in Swift 6.2 and Metal 4, rather than a wrapper around general frameworks like llama.cpp or MLX. It provides several interfaces for interaction:
- Native Mac App: A SwiftUI/AppKit application for model installation and chat.
- CLI: A command-line interface for instruction chat and raw completion.
- OpenAI-Compatible Server: A loopback server supporting Chat Completions and function tools.
- Streaming Installer: A tool that repacks Hugging Face checkpoints directly into the
.gturboformat via range requests, avoiding the need to store a full source checkpoint on disk.
Technical Requirements
To run TurboFieldfare, the following environment is required:
- Hardware: Apple Silicon Mac (arm64).
- OS: macOS 26 with Metal 4.
- Development Tools: Xcode 26 and Swift 6.2 or newer.
- Storage: Approximately 14.3 GB of free space for the model installation.
Community Insights and Analysis
Technical discussions among users highlight several key points regarding the viability of SSD-streaming for LLMs:
- Comparison to mmap: While
llama.cppcan run large models in low RAM viammap, TurboFieldfare specifically synchronizes SSD reads with inference activity to minimize latency. - SSD Wear: Users have raised questions about the impact of continuous weight reading on SSD lifespan, though no specific degradation data was provided in the source.
- Hardware Evolution: The jump in performance from M2 to M5 suggests that increasing memory bandwidth and faster SSDs are making local inference of large models increasingly practical.
"26B in 2GB is the engineering equivalent of fitting your entire apartment into a storage unit and still having room to pace"