Setting Up a Local Coding Agent on macOS with Gemma 4 and llama.cpp

Local Coding Agents on macOS: The Takeaway

Running a high-performance local coding agent on macOS is achievable by combining llama.cpp (with Metal acceleration), Gemma 4 26B, and the Pi agent harness. The key to achieving usable speeds—up to 72.2 tokens per second on an M1 Max—is the implementation of Multi-Token Prediction (MTP) speculative decoding, which provides a significant speedup over standard inference without sacrificing accuracy.

Performance Benchmarks: The Impact of MTP

Multi-Token Prediction (MTP) significantly increases generation speed. In tests conducted on an Apple M1 Max with 64 GB unified memory (macOS 15.7.7), the addition of a Q8 MTP draft model improved generation speed by approximately 24%.

Generation Speed Comparison

Setup Prompt tok/s Generation tok/s Speedup
Gemma 4 26B-A4B Q4 (llama.cpp Metal) 298.0 58.2 1.00x
Gemma 4 26B-A4B Q4 + Q8 MTP Draft 295.6 72.2 1.24x

Tuning MTP for Hardware

Performance is hardware-dependent. Testing the --spec-draft-n-max parameter (which determines the number of draft tokens) showed that a value of 3 was optimal for the M1 Max, while values above 4 led to performance degradation.

--spec-draft-n-max Prompt tok/s Generation tok/s
1 295.5 68.4
2 299.1 72.0
3 295.6 72.2
4 297.3 70.7
5 297.9 63.7
6 296.3 61.2

Runtime Comparison: llama.cpp vs. MLX

Despite MLX being specifically optimized for Apple Silicon, llama.cpp with Metal acceleration and MTP outperformed MLX-LM in generation speed for this specific model configuration.

Runtime Model Generation tok/s
llama.cpp Metal + MTP Unsloth GGUF Q4 + Q8 MTP 72.2
llama.cpp Metal Unsloth GGUF Q4 58.2
MLX-LM Unsloth UD MLX 4-bit 45.8
MLX-LM mlx-community 4-bit 43.9
MLX-LM mlx-community OptiQ 4-bit 38.1

Implementing Multimodal Support

To enable the agent to process screenshots and images, the Gemma 4 multimodal projector (mmproj-BF16.gguf) must be loaded using the --mmproj flag in llama.cpp. This allows the server to advertise multimodal support to the agent harness. Testing confirmed that loading the projector does not cause a slowdown in text generation speeds.

Step-by-Step Setup Guide

1. Install llama.cpp

Install dependencies via Homebrew and build llama.cpp with Metal and Accelerate support:

brew install cmake git tmux python@3.11

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON -DGGML_ACCELERATE=ON
cmake --build build --config Release -j

2. Download Model Files

Using the huggingface-cli, download the main model, the MTP draft model, and the multimodal projector:

pip install -U huggingface_hub hf_xet

huggingface-cli download unsloth/gemma-4-26B-A4B-it-GGUF \
  gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
  mmproj-BF16.gguf \
  MTP/gemma-4-26B-A4B-it-Q8_0-MTP.gguf \
  --local-dir models/unsloth-gemma-4-26B-A4B-it-GGUF

3. Start the Local Server

Run the llama-server to create an OpenAI-compatible endpoint at http://127.0.0.1:8080/v1:

./llama-server \
  -m models/unsloth-gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf \
  --model-draft models/unsloth-gemma-4-26B-A4B-it-GGUF/MTP/gemma-4-26B-A4B-it-Q8_0-MTP.gguf \
  --mmproj models/unsloth-gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -ngl 999 \
  -fa on \
  -c 65536 \
  --parallel 1 \
  --host 127.0.0.1 \
  --port 8080

4. Configure the Pi Agent

Add the local provider to ~/.pi/agent/models.json. Crucially, set "input": ["text", "image"] to ensure the agent utilizes the multimodal capabilities:

{
  "providers": {
    "gemma4-local": {
      "name": "Gemma 4 Local",
      "baseUrl": "http://127.0.0.1:8080/v1",
      "api": "openai-completions",
      "apiKey": "local",
      "authHeader": false,
      "models": [
        {
          "id": "gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf",
          "name": "Gemma 4 26B-A4B Q4 + MTP",
          "input": ["text", "image"],
          "contextWindow": 65536,
          "maxTokens": 8192
        }
      ]
    }
  }
}

Alternative Model: Qwen 3.6

For users prioritizing coding quality over raw speed, Qwen 3.6 35B-A3B is a viable alternative. While benchmarks suggest it is a superior coding agent, it is slower on the same hardware, yielding approximately 55 tokens per second compared to Gemma 4's 72 tokens per second.

Community Insights and Counterpoints

While the provided setup is highly optimized, community members highlighted several considerations for different hardware and software configurations:

  • Hardware Constraints: Users with 16GB RAM Macs noted that models larger than 8B are generally unusable.
  • Simplified Tooling: Several users suggested that tools like Ollama, LM Studio, or oMLX provide a more accessible entry point for beginners than building llama.cpp from source.
  • Benchmarking Validity: Some argued that short benchmarks (128 tokens) may overstate MTP speedups, as acceptance rates for predicted tokens often drop over longer generations.
  • Stability Issues: One user reported that the MTP head occasionally broke markup in certain harnesses, leading to issues with stop tokens and untidy output.

"Quickly serving me slop doesn't make it more useful. Are people really only looking at tokens per second?"

This highlights a recurring tension in the local LLM community between raw inference speed and the actual quality of the generated code.

Sources