drumih/turbo-fieldfare

What it solves

TurboFieldfare enables running the large 26-billion-parameter Gemma 4 model on Apple Silicon Macs with as little as 8 GB of RAM. Typically, such a model would require significantly more memory to load in its entirety; this project bypasses that limitation by streaming only the necessary model components from the SSD on demand.

How it works

Instead of loading the full model into memory, the runtime keeps a small core and the KV cache resident in RAM. It uses a custom Swift and Metal implementation to dynamically fetch and cache only the specific "expert" weights required for each token generation step. It employs chunked prefill to optimize prompt processing and uses a bounded cache policy to manage memory usage while streaming data from the disk.

Who it’s for

This project is for developers and researchers interested in on-device machine learning, specifically those looking to run large-scale models on resource-constrained Apple Silicon hardware. It is ideal for those who want to experiment with custom Metal kernels and memory-efficient inference techniques.

Highlights

• Runs the 26B Gemma 4 model within a ~2 GB RAM budget.
• Includes a native macOS app, a CLI, and an OpenAI-compatible local server.
• Features a streaming installer that repacks model weights directly into a custom format without needing to stage the full checkpoint.
• Built with custom Metal kernels for quantized operations, MoE (Mixture of Experts) routing, and efficient attention.
• Supports optional vision capabilities via a companion pack on M2 or newer Apple Silicon chips.