lucienhuangfu/eLLM

eLLM: Run Long-Horizon Inference Faster on CPUs Than on GPUs

What it solves

eLLM is designed to overcome the memory bandwidth bottleneck of CPUs when running Large Language Models (LLMs). While CPUs have significantly more memory capacity (DDR) than GPUs (HBM), they are slower. eLLM uses a "trade storage for computation" strategy to make CPU-based inference faster than GPUs for long-horizon tasks, such as multi-turn agent workflows and million-token context windows.

How it works

The framework optimizes the execution pipeline to reduce repeated computation and memory access:

  • Elastic Static Computation Graph: Uses a dimension-first layout to allow the same graph to support different input lengths without needing to be rebuilt.
  • Static-shape KV Cache: Replaces paged block management with preallocated fixed-shape tensors, reducing metadata overhead and improving memory access patterns.
  • Massive-dimensional Tensors: Reserves large sequence dimensions to enable full, single-pass prefill, avoiding the need to chunk prompts or reload parameters repeatedly.
  • Session Cache: Maintains the KV state across multiple turns of interaction, allowing the system to prefill only new input incrementally rather than recomputing the entire history.

Who it’s for

It is intended for developers and researchers working on long-horizon AI agents, such as computer-use agents, code copilots for large repositories, RAG systems for enterprise knowledge bases, and deep research agents that maintain state over hours or days.

Highlights

  • GPU-beating Performance: Claims to surpass GPUs in long-horizon inference and provides significant speedups over existing CPU inference frameworks like SGLang.
  • vLLM API Compatible: Plugs directly into the existing LLM ecosystem.
  • Long Context Support: Leverages TB-scale DDR memory to support near-unbounded context windows.
  • Pure CPU Inference: Requires no GPU or NPU, reducing hardware costs and energy consumption.
  • Consistent Results: Inference results are fully aligned with GPU-based outputs.

Related

  • Project
    guqiong96/Lvllm

    An integration of vLLM and lk_moe that enables large MoE models to run across CPU and GPU memory using NUMA-aware scheduling.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.

  • Project
    ztxz16/fastllm

    A high-performance LLM inference engine designed for local and server deployment, supporting mixed-device execution (GPU/CPU/Disk) and advanced optimization techniques like speculative decoding.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    casys-kaist/LLMServingSim

    A cycle-level simulator for LLM inference infrastructure that models heterogeneous hardware, disaggregated memory, and advanced serving techniques to predict performance before deployment.