lucienhuangfu/eLLM

eLLM can infer LLM on CPUs faster than on GPUs

What it solves

eLLM addresses the performance gap and high cost of running Large Language Model (LLM) inference on CPUs. While GPUs are typically the default, they often struggle with ultra-long contexts due to limited VRAM, which forces them to process prompts in small chunks. eLLM optimizes CPU inference to make it competitive—and in some long-context scenarios, faster—than multi-GPU systems by leveraging the larger memory and cache capacities of server-class CPUs.

How it works

eLLM uses a "trade storage for computation" philosophy to reduce runtime overhead and latency:

  • Elastic Static Computation Graph: It builds a globally unique static graph with a dimension-first tensor layout, allowing it to support various input lengths without needing to rebuild the graph.
  • Static-Shape KV Cache: Instead of paged memory management, it preallocates a fixed-shape tensor for the Key-Value (KV) cache, enabling direct coordinate-based access and reducing metadata overhead.
  • Massive-Dimensional Tensors: By reserving very large dimensions for tokens and sequences, it enables "unbounded" KV tensors that can handle full Prefill in a single pass.
  • Head-by-Head Attention: During the Prefill stage, it computes attention one head at a time. This keeps a single head's KV data resident in the CPU's large L3 cache for as long as possible, minimizing repeated memory loads.

Who it’s for

It is designed for users running long-context workloads on CPU servers (Intel Xeon 4th Gen+ with AMX support), specifically for:

  • AI Agents: Computer-use agents that ingest large contexts for reasoning.
  • Developers: Code Copilots handling cross-file or cross-module contexts.
  • Enterprise RAG: Systems injecting large external documents into prompts.
  • Researchers: Deep research workflows involving multi-step synthesis of vast amounts of material.

Highlights

  • Pure CPU Inference: No GPU or NPU required; runs on Xeon/EPYC servers.
  • vLLM API Compatible: Integrates easily with the existing LLM ecosystem.
  • Long Context Support: Leverages massive CPU RAM to support near-unbounded context windows.
  • Reduced Latency: Optimizes Time to First Token (TTFT) by avoiding the chunked processing required by GPUs.
  • Cost Effective: Significantly lower hardware and per-user inference costs compared to GPU deployments.