drumih/turbo-fieldfare

TurboFieldfare – on‑device inference for Gemma 4 26B on Apple Silicon

What it is

• A Swift + Metal runtime that lets you run the 26‑billion‑parameter Gemma 4 26B‑A4B instruction‑tuned model on any Apple‑silicon Mac, even those with only 8 GB of RAM.
• It does not load the full 14 GB checkpoint into memory. Instead it keeps a 1.35 GB shared core and the FP16 KV cache resident, and streams the needed MoE “expert” weights from SSD on‑the‑fly.
• The project ships a native macOS app, a CLI, a Swift library, and a loop‑back OpenAI‑compatible server, all built on top of custom Metal kernels.

Why it matters

• Large language models usually need tens of gigabytes of RAM; TurboFieldfare squeezes inference down to ~2 GB, making 26 B‑scale models usable on the base‑spec M2 MacBook Air.
• It demonstrates a practical “expert‑cache‑streaming” technique that could be reused for other mixture‑of‑experts (MoE) models on limited‑memory devices.

Key components

Component	Role
TurboFieldfare (Swift library)	Core runtime + Metal kernels for quantized GEMV, attention, MoE routing, sampling, etc.
TurboFieldfareMac	SwiftUI/AppKit GUI that downloads, repacks, loads the model and lets you chat with it.
TurboFieldfareCLI	Command‑line chat/completion tool that works with the same .gturbo model directory.
TurboFieldfareServer	Local OpenAI‑compatible HTTP server (loopback only) supporting chat, streaming, and function‑tool calls.
TurboFieldfareRepack	Streaming installer that pulls only the required byte‑ranges from Hugging Face, writes them directly into the custom .gturbo layout, and verifies the install.
TurboFieldfareDecodeService	Small helper process that owns the Metal context for the GUI.

How to get started

1. Clone & build

   git clone https://github.com/drumih/turbo-fieldfare.git
   cd turbo-fieldfare
   swift build -c release   # builds the app + decode service
   .build/release/TurboFieldfareMac
   

2. Download the model – the app (or TurboFieldfareRepack) streams ~15 GB from the pinned Hugging Face checkpoint and creates a .gturbo directory (scratch/gemma4.gturbo).
3. Load & generate – press Load Model, type a prompt, and hit Generate (or use the CLI with TurboFieldfareCLI).
4. Optional vision support – run TurboFieldfareRepack with --vision-output to add a ~1.1 GB image‑tower pack; the app/CLI then accept --image arguments.

Supported hardware & software

• Apple‑silicon Macs (ARM64) – tested on an 8 GB M2 MacBook Air; also works on M5, M4, etc.
• macOS 26 + Metal 4 (Xcode 26, Swift 6.2 or newer).
• Requires ~15 GB free storage for the model and ~2 GB RAM during inference.

Performance snapshot (from the repo’s benchmark table)

• M2 (8 GB): 5.1 – 6.3 tokens / second decode.
• M5 (24 GB): 31 – 35 tokens / second decode. Community members can submit their own numbers via the provided benchmark guide.

Usage details

• Prompting – defaults: temperature 0.2, top‑k 64, top‑p 0.95. Use --temperature 0 for deterministic output.
• KV cache – FP16, 4 K token window for sliding‑window layers; full‑attention layers use linear storage.
• Image input – when the vision pack is installed, you can send a single image with --image <file> (CLI) or attach it in the GUI; multi‑turn image chats are supported via a JSON messages file.
• OpenAI server – runs on http://127.0.0.1:8080/v1; supports chat completions, streaming, and function‑tool calls (client must authorize tool execution).

Development & contribution

• Run the test suite with Scripts/test.sh.
• Follow the “community benchmark guide” to add performance results.
• The project is Apache 2.0 licensed; model weights are downloaded separately and remain under the original Hugging Face terms.

Current scope & future plans

• In‑scope: text generation (Gemma 4 26B‑A4B) and optional vision tower on Apple‑silicon Macs.
• Out‑of‑scope: audio/video modalities, remote server deployment, non‑Apple hardware.
• Future work: iPhone/iPad apps and broader benchmarking across more Apple‑silicon devices.

---

TL;DR: TurboFieldfare is a Swift/Metal‑based inference engine that makes the 26 B‑parameter Gemma 4 model run on low‑memory Apple‑silicon Macs by streaming MoE experts from SSD. It provides a GUI, CLI, and local OpenAI‑compatible server, all under an Apache 2.0 license.