Running Gemma 4 26B on Legacy Hardware: Ivy Bridge Xeon Case Study

Executive Summary

It is possible to run Google's Gemma 4 26B mixture-of-experts (MoE) model at approximately 5 tokens per second on 13-year-old enterprise hardware without a GPU. This was achieved by identifying and patching a silent failure in the ik_llama.cpp inference engine, specifically addressing the lack of AVX2 and FMA3 instruction sets on Intel Ivy Bridge (v2) CPUs.

Hardware and Performance Specifications

Running a modern 26-billion parameter model on legacy silicon requires specific hardware configurations and results in the following performance metrics:

• Hardware: Repurposed HP StoreVirtual server with dual Intel Xeon E5-2690 v2 (Ivy Bridge, 2013) CPUs and DDR3 RAM.
• Instruction Set: AVX1 only (No AVX2, No FMA3).
• Model Configuration: Gemma 4 26B-A4B (MoE), quantized to Q8_0.
• Decode Speed: ~5.2 tokens/sec.
• Prompt Evaluation: ~16 tokens/sec.
• Hardware Cost: Under $300.

Technical Analysis of the AVX2 Compatibility Bug

While many inference engines provide scalar fallbacks, ik_llama.cpp (a fork of llama.cpp optimized for Gemma 4) assumed AVX2 as the minimum requirement for certain critical paths. This led to two distinct failure modes on Ivy Bridge hardware.

The Build Failure

Initial attempts to compile the engine failed because the scalar #else branches in iqk_quantize.cpp still referenced AVX2 helpers like hsum_i32_8. To resolve this, these paths were rewritten as portable scalar loops, and #if GGML_USE_IQK_MULMAT guards were added to prevent stray IQK calls from leaking into ggml.c and ggml-quants.c.

The Silent Runtime Failure

Even after successful compilation, the model produced "fluent-looking multilingual gibberish." This was caused by a mismatch between the graph builder and the compute dispatcher:

1. The Cause: The Gemma 4 MoE feed-forward network emits MOE_FUSED_UP_GATE and FUSED_UP_GATE operations. While the compute dispatcher's switch case for these ops was gated behind GGML_USE_IQK_MULMAT (and thus disabled on AVX1), the graph builder continued to emit them unconditionally.
2. The Result: The dispatcher had no case for these enums, causing them to fall through to a default state. Consequently, the destination tensors for every expert FFN were never computed, leaving the hidden state filled with uninitialized memory.
3. The Diagnosis: The bug was identified by instrumenting raw logits before sampling. A mean logit of +16 (when it should be near zero) indicated that the residual stream was being pushed by uninitialized memory containing small positive floats.

The Implementation Fix

The solution, submitted as PR #2138 to ik_llama.cpp, avoids touching the dispatcher and instead modifies the graph builder to emit operations that have existing non-IQK compute paths:

• MoE Layers: When GGML_USE_IQK_MULMAT is off, the up_gate_exps tensor is split into two ggml_view_3d slices. The engine then runs two separate ggml_mul_mat_id calls and combines them using ggml_fused_mul_unary(gate, up, SILU).
• Dense Layers: The ggml_fused_up_gate operation is treated with the same split-and-combine logic.
• CI Integration: Stubs in the iqk sources were updated to match iqk_mul_mat.h signatures to allow the test suite to build on non-AVX2 hardware.

Deployment Constraints and Trade-offs

Users attempting to reproduce these results on pre-AVX2 hardware must observe the following constraints:

• Disable Run-time Repacking: The --run-time-repack flag must be omitted. This flag reorders quantized weights into a Q8_0_R8 interleaved layout that is AVX2-only; using it on AVX1 hardware results in garbled output.
• Build Flag: The project must be compiled with GGML_USE_IQK_MULMAT turned off.

Economic and Efficiency Considerations

Community discussion highlights a significant gap between the feasibility of local inference and its economic efficiency. While the hardware cost is low, the operational cost is high:

"I estimate that the server consumes probably around 500W during inference... 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider."

Critics note that while local inference is essential for privacy and data sovereignty, it is rarely a cost-saving measure compared to modern API providers using H100/H200 GPUs, which offer significantly higher throughput and lower energy costs per token.