lucienhuangfu/eLLM

eLLM: Run Long-Horizon Inference Faster on CPUs Than on GPUs

What it solves

eLLM is designed to overcome the memory bandwidth bottleneck of CPUs when running Large Language Models (LLMs), specifically for long-horizon tasks. It enables CPU servers to perform inference faster than GPUs in scenarios involving million-token context windows and multi-turn interactions, reducing the need for expensive, power-hungry GPU hardware.

How it works

The framework adopts a "trade storage for computation" strategy, leveraging the large capacity of DDR memory to minimize repeated calculations. Key technical implementations include:

  • Elastic Static Computation Graph: Uses a dimension-first layout to allow the same execution graph to support varying input lengths without needing to be rebuilt.
  • Static-shape KV Cache: Replaces paged block management with preallocated fixed-shape tensors to reduce metadata overhead and avoid cache misses.
  • Massive-dimensional Tensors: Reserves large sequence dimensions to support full Prefill and avoid repeated parameter loading for ultra-long prompts.
  • Session Cache: Maintains KV state across multi-turn interactions, allowing the system to only Prefill new input incrementally rather than recomputing the entire history.

Who it’s for

It is primarily intended for developers and researchers working on long-horizon AI agents, such as computer-use agents, code copilots for large repositories, RAG systems for enterprise knowledge bases, and deep research agents that maintain state over hours or days.

Highlights

  • GPU-surpassing performance: Claims to outperform GPUs in long-context inference and long-horizon tasks.
  • vLLM API compatibility: Plugs directly into the existing LLM ecosystem.
  • Ultra-long context: Supports million-token, near-unbounded context windows using TB-scale memory.
  • Pure CPU inference: Requires no GPU/NPU, reducing infrastructure costs and energy consumption.
  • Consistent results: Inference outputs remain aligned with GPU-based results.

Related

  • Project
    xLLM-AI/xllm

    xLLM is an open‑source C++ inference framework that delivers high‑throughput, low‑latency LLM serving on Chinese AI accelerators (Ascend, Cambricon, Moore Threads, etc.). It separates a brpc‑based service layer from a hardware‑optimized engine, includes hybrid KV‑cache management via Mooncake, and provides ready‑to‑run deployment scripts for models such as GLM‑5.3‑Flash and MiniMax‑M3. The project is Apache‑2.0 licensed and backed by JD.com and several Chinese university labs.

  • Project
    guqiong96/Lvllm

    An extension of vLLM that enables hybrid GPU and NUMA-aware CPU inference for MoE models, allowing them to span across VRAM and system memory.

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without a server.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.

  • Project
    snowflakedb/ArcticInference

    An open-source vLLM plugin that optimizes LLM and embedding inference to increase throughput and reduce latency using advanced parallelism and speculative decoding.