Running Gemma 4 26B on Legacy Xeon Hardware

Google's Gemma 4 26B mixture-of-experts (MoE) model can run at approximately 5 tokens per second on 13-year-old Ivy Bridge Xeon CPUs without a GPU. This is achieved by implementing scalar fallbacks for AVX2 and FMA3 instruction sets within the ik_llama.cpp inference engine, enabling modern MoE architectures to operate on legacy enterprise hardware.

Hardware and Performance Benchmarks

Running a 26-billion-parameter model on hardware from 2013 demonstrates that memory bandwidth, rather than raw compute instructions, is the primary bottleneck for local LLM inference on older systems.

System Specifications:

• Hardware: Repurposed HP StoreVirtual storage box
• CPU: Dual Intel Xeon E5-2690 v2 (Ivy Bridge, 2013)
• Instruction Sets: AVX1 only (No AVX2, No FMA3)
• RAM: DDR3
• GPU: None
• Total Hardware Cost: Under $300

Performance Metrics (Gemma 4 26B-A4B, Q8_0):

• Decode Speed: ~5.2 tokens/sec
• Prompt Evaluation: ~16 tokens/sec

Technical Challenge: The AVX2 Instruction Gap

Modern high-performance inference kernels, including those in the ik_llama.cpp fork, typically assume a minimum hardware floor of AVX2 and FMA3 (introduced with Intel's Haswell/v3 generation in 2014). Ivy Bridge (v2) CPUs lack these instructions, leading to immediate build failures or runtime crashes when the software attempts to execute non-existent CPU instructions.

While disabling GGML_USE_IQK_MULMAT at build time forces most of the codebase to fall back to scalar or SSE math, certain graph operations in the Gemma 4 MoE feed-forward network (FFN) remained problematic. Specifically, the MOE_FUSED_UP_GATE and FUSED_UP_GATE operations were being emitted by the graph builder but lacked corresponding compute paths in the dispatcher for non-AVX2 builds. This resulted in a silent failure where destination tensors for every expert FFN were never computed, leaving the model to process uninitialized memory.

The Solution: Implementing Scalar Fallbacks

To resolve the silent corruption and enable compatibility with pre-AVX2 hardware, three primary modifications were implemented in a patch (submitted as PR #2138 to ik_llama.cpp):

1. Correcting Scalar Compile Paths

Several scalar #else branches in iqk_quantize.cpp incorrectly referenced AVX2 helpers like hsum_i32_8. These were rewritten as portable scalar loops, and #if GGML_USE_IQK_MULMAT guards were added to prevent AVX2-specific IQK calls from leaking into the build.

2. Fixing the Runtime Dispatcher Bug

Instead of modifying the dispatcher, the graph builder was updated to emit operations that have existing compute paths on legacy hardware. When GGML_USE_IQK_MULMAT is disabled:

• The combined up_gate_exps tensor is split into two ggml_view_3d slices.
• Two separate ggml_mul_mat_id calls are executed.
• The results are combined using ggml_fused_mul_unary(gate, up, SILU).

This approach leverages stock GGML implementations (mul_mat_id and fused_mul_unary), ensuring that the MoE layers compute correctly without requiring AVX2 fused kernels.

3. CI and Build System Stubs

To ensure the test suite could run on non-AVX2 hardware, missing <cstdint> includes and incorrect function signatures in the #else stub sections of the IQK sources were corrected to match iqk_mul_mat.h.

Deployment Constraints and Caveats

Users attempting to reproduce these results on pre-AVX2 hardware must observe the following constraints:

• Disable Run-Time Repacking: The --run-time-repack flag must be omitted. This feature reorders quantized weights into an interleaved layout (Q8_0_R8) that is AVX2-only; using it on AVX1 hardware results in garbled output.
• Build Configuration: The project must be compiled with GGML_USE_IQK_MULMAT set to off.

Community Perspectives on Local Legacy Inference

Discussion among technical users highlights a tension between the educational/privacy value of local inference and the economic reality of cloud providers.

"I estimate that the server consumes probably around 500W during inference... 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider."

While the monetary cost of electricity for legacy hardware may exceed the cost of API tokens, proponents argue that the primary drivers for this setup are data privacy and the ability to maintain a local fallback when paid APIs are unavailable.