kvcache-ai/ktransformers

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

What it solves

Large Mixture-of-Experts (MoE) models such as DeepSeek-V3/R1 require massive GPU memory (often over 80GB) for inference or fine-tuning, which is out of reach for typical consumer setups. KTransformers tackles this by splitting computation between CPU and GPU, placing frequently used 'hot' experts on the GPU and offloading the rest to the CPU, using highly optimized kernels to keep CPU execution fast enough.

Related

  • Project
    guqiong96/Lvllm

    An integration of vLLM and lk_moe that enables large MoE models to run across CPU and GPU memory using NUMA-aware scheduling.

  • Project
    ztxz16/fastllm

    A high-performance LLM inference engine designed for local and server deployment, supporting mixed-device execution (GPU/CPU/Disk) and advanced optimization techniques like speculative decoding.

  • Dispatch
    Mixture of Experts (MoE) Explained

    Mixture of Experts (MoE) allows transformer models to scale parameters while maintaining efficient pretraining and faster inference by activating only a subset of neural network experts per token.

  • Project
    Helldez/BigMoeOnEdge

    A runtime engine that enables running Mixture-of-Experts models larger than a device's RAM by streaming experts from flash storage, allowing massive LLMs to run on smartphones and CPUs.

  • Project
    Soulmate-Halo/heterogeneous-gpu-pd-lab

    The repository is a genuine software project that provides Docker images, deployment guides, and raw benchmark data for heterogeneous GPU inference of large mixture‑of‑experts LLMs (e.g., Qwen‑3.8‑Flash‑Next, DeepSeek‑V4.1‑Flash). It shows how to split prefill and decode across different GPU types (RTX 3080, RTX 6000D, DGX Spark, AI Max 395) and reports untuned performance numbers up to 16 k tok/s prefilling and ~442 tok/s decoding.