Benchmarking Nine Coding Harnesses on a MacBook Pro with Local Qwen 3.8 27B Model

TL;DR

Running nine popular coding‑agent harnesses on a single M4 MacBook Pro with a locally‑served Qwen 3.8 27B model reveals three clear performance groups: (1) lean, stable harnesses (pi, mini‑swe‑agent, chad) deliver ~8 tokens / s with sub‑second turn latency; (2) heavy but disciplined harnesses (dsh, cline, codex, goose) achieve 6–8 tokens / s but incur longer first‑token waits; (3) heavy‑to‑start harnesses (crush, opencode) spend 3–4 minutes before any output and fall to ~5–6 tokens / s. The differences stem primarily from system‑prompt size, tool‑schema count, and cache‑reuse efficiency.

Experimental Setup

  • Hardware: Apple M4 MacBook Pro, 24 GB RAM, macOS 26.6.2.
  • Model: Qwen 3.8 27B, 3‑bit quantized via unsloth/Qwen3.8-27B-GGUF, served by llama.cpp (build 10470).
  • Server: Single llama-server instance shared across all harnesses; a proxy enforced a uniform sampling regime (temperature = 1.0, top_k = 20, top_p = 0.95, min_p = 0.05).
  • Cache: 32,768‑token unified prefix cache, split across four slots.
  • Benchmark: Eight Exercism Python exercises, each run in auto‑approve mode with an identical one‑sentence prompt. Metrics were collected from the server’s own accounting (except for the two chad rows, which used internal traces).

Metric Definitions

Metric Meaning
wait before 1st token Time to prefill the system prompt, tool schemas, and the first user request.
wait / later turn Median and 90th‑percentile pause after the first turn (excluding side‑requests).
cache reuse Percentage of tokens served from the prefix cache on subsequent turns.
experienced tokens/second Total generated tokens divided by wall‑clock time, including prefill and tool overhead.
pass (gate) Number of Exercism tasks completed within a 1,200 s timeout.

Why Local Inference Is Harder Than Cloud

  1. Large system prompts & tool schemas – On a laptop that reads ~90 tokens/s and writes ~10 tokens/s, a 2,000‑token prompt costs ~22 s before any generation; an 18,000‑token prompt (as used by Opencode) costs ~226 s. Cloud GPUs prefill at >10k tokens/s, collapsing these delays to fractions of a second.
  2. Reduced context window – After consuming the system prompt, the remaining context is often <32 k tokens. Opencode’s 18k‑token prompt leaves only ~44 % of the window for actual work, whereas the lean pi harness retains ~94 %.
  3. Side‑request overhead – Harnesses that repeatedly issue auxiliary requests (e.g., for code summaries) cause the local model to queue or re‑prefill, effectively making the GPU “busy” >100 % of wall‑clock time.

Benchmark Results

Harness Version Tools Prompt (tokens) 1st‑token wait Later‑turn wait (median·p90) Cache reuse Tokens / s Pass (out of 24)
mini‑swe‑agent 2.4.6 1 1,171 12.2 s 3.6 s·21 s 96 % 8.0 11 (14 timeouts)
pi 0.80.3 4 2,008 21.6 s 1.3 s·22 s 99 % 8.1 19 (7 timeouts)
cline 3.0.60–61 26 5,876 64.1 s 9.9 s·52 s 94 % 7.3 17
codex 0.151.0 10 7,804 87.8 s 9.6 s·28 s 94 % 6.9 19 (5 timeouts)
dsh 0.1.1‑rc.2 25 8,052 94.4 s 2.2 s·34 s 99 % 7.2 18
goose 1.50.0 18 9,617 110.3 s 1.0 s·22 s 100 % 8.0 22 (3 timeouts)
crush 0.92.0 26 16,263 199.8 s 1.8 s·40 s 100 % 5.8 18 (8 timeouts)
opencode 1.17.12 10 18,046 225.7 s 4.6 s·44 s 99 % 5.7 15 (13 timeouts)
chad (llama.cpp) 2.0.3 5 2,563 25.6 s 0.8 s·19 s 99 % 7.9 24
chad (MLX, serial) * 2.0.3 5 2,566 4.7 s 1.0 s·19 s 99 % 12.4 21 (3 timeouts)
chad (MLX, dflash2) * 2.0.3 5 2,562 4.6 s 0.9 s·36 s 99 % 17.4 22 (3 timeouts)

Interpretation of the Numbers

  • Lean harnesses (pi, mini‑swe‑agent, chad) keep the system prompt under ~2k tokens, achieve >99 % cache reuse, and stay within ~8 tokens/s. The MLX variants of chad double the throughput (up to 17.4 tokens/s) by owning the cache in‑process.
  • Heavy but disciplined harnesses suffer longer first‑token waits (up to ~110 s) but maintain high cache reuse, so their steady‑state throughput remains respectable (≈7 tokens/s).
  • Heavy‑to‑start harnesses (crush, opencode) spend >3 minutes before any output and drop to ≤6 tokens/s, making them impractical on laptops.
  • Pass rates correlate with latency: only chad (both variants) solved all 24 tasks; the next best was goose (22/24). Mini‑swe‑agent’s low pass count stems from frequent timeouts despite decent token speed.

Design Lessons for Local‑Model Harnesses

  1. Trim the system prompt – Every extra token adds ~0.01 s of prefill latency on a laptop. Aim for <2k tokens.
  2. Limit tool schemas – Each additional tool adds to the prompt size and reduces the usable context window.
  3. Persist the prefix cache – Re‑using the cached prefill across turns (≥95 % reuse) eliminates repeated work and dramatically improves experienced throughput.
  4. Co‑locate the agent loop and model – chad’s in‑process MLX implementation shows that owning the cache removes a network round‑trip, yielding a 2×‑3× speedup.
  5. Provide a drafter – The DFlash2 drafter raised chad’s token rate from 12.4 → 17.4 tokens/s, demonstrating the value of a lightweight draft model for early token generation.

Community Feedback Highlights

"If you need a coding agent for resource‑constrained environments, check out hax – a 0.7 MB native binary with a minimalist prompt and tool set." – OleksandrC

"Benchmarks like this change rapidly; a reproducible repo would let anyone compare per‑turn token counts, first‑token latency, and pass rates on their own hardware." – humbleferret

"jcode beats everything in RAM usage with a tiny Rust binary, yet it wasn’t included in the study." – lrvick

"Reasonix may be an interesting comparison because they focus heavily on prefix‑cache reuse." – swiftcoder

These comments reinforce the importance of lightweight prompts, open‑source reproducibility, and the existence of alternative ultra‑lean agents not covered in the original matrix.

Reproducing the Benchmark

The entire experiment is scripted in the chad repository:

# Clone the repo
git clone https://github.com/nathansutton/chad.git
cd chad

# Install dependencies (uv recommended)
uv run python benchmarks/matrix/run.py setup   # install models, build llama.cpp
uv run python benchmarks/matrix/run.py smoke   # sanity‑check
uv run python benchmarks/matrix/run.py llama   # run all harnesses via llama.cpp server
uv run python benchmarks/matrix/run.py mlx     # run chad with MLX back‑ends
uv run python benchmarks/matrix/run.py table   # generate the markdown table shown above

All raw data (grid.json, turns.jsonl) are committed alongside the benchmark code, ensuring full traceability from a single row to the aggregated numbers.

Takeaway

When swapping a cloud LLM for a local model on a laptop, the dominant factor is prompt‑size induced prefill latency. Harnesses that were designed for virtually free cloud prefill (large system prompts, many tool schemas) become unusable locally. A disciplined, minimal prompt combined with persistent prefix caching and, if possible, an in‑process model loop can deliver near‑cloud‑like throughput on consumer hardware.

Sources

Related