Breaking the Memory Wall: Running Gemma 4 on a 10-Year-Old Xeon

The prevailing narrative in local AI is that you need the latest H100s or a maxed-out Apple Silicon Mac to run state-of-the-art models. However, the real bottleneck for Large Language Model (LLM) inference isn't always raw compute—it's the "memory wall." When generating text, the system must haul gigabytes of weights from RAM into the CPU cache for every single token produced. On aging hardware, this process is agonizingly slow.

In a recent technical experiment, a 2016-era Intel Xeon E5-2620 v4 server with 128GB of DDR3 RAM and absolutely no GPU was used to run Gemma 4 (a 26B parameter Mixture-of-Experts model). By bypassing "black-box" tools like Ollama and diving into the granular optimization flags of the ik_llama.cpp fork, it was possible to achieve "reading speed" generation on hardware that predates the very architecture it is running.

The Hardware Constraints

To understand the challenge, one must look at the specifications of the machine in question:

  • CPU: Intel Xeon E5-2620 v4 @ 2.10 GHz (8 physical cores, 16 threads)
  • Instruction Sets: AVX2 (No AVX-512, No BF16)
  • Memory: 128 GB DDR3 (significantly slower than modern DDR4/DDR5)
  • GPU: None

In this environment, the CPU is rarely the bottleneck; the memory bus is. The processor often sits idle, waiting for weights to move from the slow DDR3 RAM into the L3 cache. To overcome this, every available software lever must be pulled.

Squeezing Performance from the CPU

Running a model of this size on a CPU requires a precise combination of optimizations to minimize memory movement and maximize cache utilization.

Speculative Decoding

One of the most effective ways to bypass the memory wall is speculative decoding. By pairing the 26B "verifier" model with a much smaller "drafter" model, the system can guess several tokens at once.

Because the drafter is tiny, its active layers often fit entirely within the CPU's L3 cache. The verifier then checks these tokens in a single pass. This is significantly more efficient on CPUs than GPUs because CPU compute is relatively cheap compared to the high cost of streaming the verifier's massive weights through the cache.

Optimizing Mixture-of-Experts (MoE)

Gemma 4 26B-A4B uses a MoE architecture with 128 experts, only 8 of which are active per token. On a CPU, jumping between these experts can cause "cache thrashing," where the CPU constantly dumps its cache to fetch new weights from RAM.

Two key optimizations mitigate this:

  1. --cpu-moe: Tunes the routing to be smarter about expert selection, keeping weights in the local cache longer.
  2. --merge-up-gate-experts: Fuses two per-expert projections into a single matrix multiplication (matmul). This reduces the number of trips across the memory bus, effectively combining two operations into one.

Memory Management and Pinning

To prevent the operating system from sabotaging performance, memory pinning is essential. The --mlock flag tells the Linux kernel to pin the model weights strictly in physical RAM, preventing the OS from "swapping" data to the hard drive—an event that would cause generation speed to drop to zero.

Additionally, --run-time-repack reorganizes weight matrices in RAM during startup to align perfectly with the CPU's cache layout, reducing "cache misses" during inference.

The "Dark Arts" of Graph Layout and Attention

Beyond standard flags, advanced memory allocation and custom kernels provide the final performance boost.

Graph vs. Layer Splitting

While the engine attempted to use "Graph Split" (vertical slicing of the computational graph to keep all hardware running at 100%), the complexity of the MTP (Multi-Token Prediction) architecture in Gemma 4 forced a fallback to "Layer Split" (horizontal slicing). Despite this, using --split-mode-f32 ensures that intermediate connection points use 32-bit precision, preventing intelligence loss or hallucinations due to rounding errors.

CPU-Based Flash Attention

Perhaps the most significant achievement is the porting of Flash Attention to standard CPUs. Normally a GPU-only feature, Flash Attention avoids "materializing" the massive $N \times N$ attention matrix in RAM. Instead, it calculates attention scores in small chunks and consumes them entirely within the processor's local cache.

Combined with Multi-Head Latent Attention (--mla-use 3), which compresses the KV cache (the model's short-term memory), the system can handle massive contexts without exhausting system RAM.

Results and Implications

The final configuration resulted in a memory footprint of approximately 82 GB (25 GB for weights and 56 GB for the KV cache at a 262K context). While the author describes the speed as "reading speed," community discussions suggest this translates to roughly 8–12 tokens per second.

The Trade-offs

While technically impressive, this approach isn't without drawbacks. As noted in the community discussion:

"Depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all... and these servers are LOUD."

Furthermore, the complexity of the setup—requiring 25 specific flags, many of which are undocumented—highlights a significant "usability moat." Most users rely on black-box wrappers that hide these levers, often sacrificing massive amounts of performance for the sake of a simpler UI.

Conclusion

The ability to run a 26B parameter model on a decade-old server proves that the bottleneck to local AI is not solely the silicon; it is the understanding of the inference engine. By treating the deployment pipeline as a serious engineering task and mapping the model architecture directly to the physical hardware, the barriers to entry for high-performance local AI are significantly lowered. The "bleeding edge" is not just for those with the latest GPUs—it is available to anyone willing to get their hands dirty with the command line.

Sources