Running Gemma 4 26B on a 13‑year‑old Xeon at 5 Tokens/sec (CPU‑only)

TL;DR

A 13‑year‑old HP StoreVirtual box with two Ivy Bridge Xeon E5‑2690 v2 CPUs (AVX1 only) can generate text with Google’s Gemma 4 26 B‑parameter mixture‑of‑experts model at roughly 5 tokens / second using a patched, CPU‑only build of ik_llama.cpp. The fix removes AVX2 dependencies and adds fallback implementations for two MoE graph ops that were silently failing on pre‑AVX2 silicon.

Hardware context

  • Server: Repurposed HP StoreVirtual storage appliance (≈2013), dual Xeon E5‑2690 v2 (Ivy Bridge), DDR3 memory, no GPU.
  • Instruction set: AVX1 only; lacks AVX2 and FMA3 introduced with Haswell (v3) in 2014.
  • Cost: Under $300 for the whole box.
  • Performance: ~5.2 tokens / sec decode, ~16 tokens / sec prompt evaluation.

Why this matters

Running a modern 26 B‑parameter MoE model on such legacy hardware demonstrates that:

  1. Local inference is possible without expensive GPUs, useful for privacy‑sensitive workloads or when cloud costs spike.
  2. CPU‑only inference can be made robust by addressing instruction‑set mismatches and hidden bugs, extending the useful life of old enterprise hardware.
  3. Open‑source tooling (the ik_llama.cpp fork) can be adapted to older CPUs, widening accessibility of large language models.

The original obstacle

A viral Hacker News post showed Gemma 4 running on a 2016 Broadwell Xeon using ik_llama.cpp with many performance tricks (speculative decoding, flash attention, MoE routing, runtime weight repacking). When the author tried the same on an Ivy Bridge box, the build failed because the fork assumes AVX2. The missing AVX2 instructions caused compile‑time errors and, after a quick workaround to disable the GGML_USE_IQK_MULMAT flag, the model still produced nonsensical multilingual output.

The silent failure

Two MoE graph operations—MOE_FUSED_UP_GATE and FUSED_UP_GATE—were still emitted by the graph builder even when GGML_USE_IQK_MULMAT was disabled. The dispatcher had no case for these ops on non‑AVX2 builds, so the tensors for every expert’s feed‑forward network remained uninitialized. The result was deterministic gibberish with uniformly high logits (mean ≈ +16), not random noise.

The fix (PR #2138)

The patch consists of three main parts, all gated behind #if !GGML_USE_IQK_MULMAT so AVX2 builds remain unchanged.

  1. Compile‑time fixes – Portable scalar loops replace stray AVX2 calls in iqk_quantize.cpp; missing includes and mismatched function signatures are corrected, allowing the code to compile on AVX1 CPUs.
  2. Runtime bug fix – The graph builder now emits ops that have valid scalar implementations:
    • For MoE up‑gate, the combined weight tensor is split into separate gate and up slices, each multiplied with ggml_mul_mat_id, then combined with ggml_fused_mul_unary (SwiGLU). The dense analog receives the same treatment.
    • These paths use existing non‑IQK kernels (mul_mat_id and fused_mul_unary), preserving correctness without AVX2.
  3. CI stubs – Stub functions in the IQK sources are synchronized with headers, fixing missing <cstdint> includes and signature mismatches that prevented the test suite from linking.

The patch does not address the --run-time-repack flag, which still reorders weights into an AVX2‑only layout; the recommended workaround is to drop that flag.

Reproducing the result

  1. Hardware: Dual Xeon E5‑2690 v2 (AVX1), DDR3, no GPU.
  2. Build: Clone the branch referenced in PR #2138, compile with GGML_USE_IQK_MULMAT=0.
  3. Model: Download Gemma 4 26B‑A4B, quantized to Q8_0.
  4. Run: Use the standard ik_llama.cpp CPU flags, omit --run-time-repack.

You should observe ~5 tokens / sec decode on a CPU‑only system.

Community reactions

  • Performance expectations: Several commenters note that while 5 tps is usable for low‑throughput tasks, the prompt‑evaluation speed (~16 tps) can be a bottleneck for interactive use.
  • Cost comparison: One analysis estimates that running the box (~500 W) costs about $0.15 / hour in Germany, roughly 30× more than the $0.005 / hour token cost from cloud providers for the same output volume.
  • Future outlook: A user predicts that by mid‑2027, >200 B‑parameter MoE models may run on consumer hardware, citing a 35 B‑parameter model achieving 7‑9 tps on a 16 GB MacBook Air.
  • Other hardware: Users report similar speeds on 2013 Mac Pros and dual‑Xeon servers with more RAM, confirming the approach scales across comparable legacy boxes.

Why publish this on a company blog?

The author argues that the real value lies not in the cheap hardware itself but in the skill set required to:

  • Diagnose low‑level CPU instruction mismatches.
  • Navigate and patch performance‑critical C++ code.
  • Iterate with an AI assistant (Claude) to produce a correct fix.

These capabilities are analogous to maintaining legacy web apps or databases: the ability to “dig until you find where the leverage is” is a marketable service.

Takeaways

  • Legacy CPUs can run modern MoE models if you provide proper AVX1 fallbacks for critical kernels.
  • Silent bugs (missing dispatcher cases) can produce deterministic but incorrect output; instrumenting logits is an effective debugging step.
  • Open‑source patches enable the community to extend the life of old hardware, offering a low‑cost fallback for token‑based services.

The patch is currently open on GitHub at https://github.com/ikawrakow/ik_llama.cpp/pull/2138 and awaiting maintainer review.

Sources

Related