kvcache-ai/ktransformers

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

What it solves

KTransformers addresses the high hardware requirements for running and fine-tuning ultra-large Mixture-of-Experts (MoE) models. It allows these models to run on consumer-grade hardware by utilizing CPU-GPU heterogeneous computing, reducing the need for massive amounts of expensive GPU VRAM.

How it works

The framework employs a hybrid computing approach where different parts of the model are placed on different hardware. It uses "expert scheduling" to keep frequently used "hot" experts on the GPU while offloading "cold" experts to the CPU. To maintain performance, it utilizes optimized kernels for Intel AMX and AVX512/AVX2 for quantized inference and integrates with LlamaFactory to enable efficient fine-tuning (SFT) using techniques like LoRA and block-FP8, which can be significantly faster than traditional ZeRO-Offload methods.

Who it’s for

It is designed for researchers and developers who want to run or fine-tune cutting-edge large models (like DeepSeek-V3/R1 or GLM) on limited hardware, such as consumer GPUs or mixed CPU-GPU server setups.

Highlights

  • Heterogeneous Computing: Supports CPU-GPU hybrid inference and fine-tuning to lower VRAM barriers.
  • Hardware Acceleration: Optimized for Intel AMX, AVX512, AVX2, and supports ROCm (AMD) and Ascend NPUs.
  • MoE Optimization: Specialized memory management and expert scheduling for Mixture-of-Experts models.
  • Fine-Tuning Integration: Seamlessly integrates with LlamaFactory for LoRA and full-parameter fine-tuning of ultra-large models.

Related

  • Project
    guqiong96/Lvllm

    An extension of vLLM that enables hybrid GPU and NUMA-aware CPU inference for MoE models, allowing them to span across VRAM and system memory.

  • Project
    ztxz16/fastllm

    A high-performance C++ LLM inference engine that enables hybrid deployment of dense and MoE models across GPU, CPU, NUMA, and disk to maximize hardware utilization.

  • Project
    EfficientMoE/MoE-Infinity

    A cost-effective inference library that enables running large Mixture-of-Experts models on memory-constrained GPUs by offloading expert weights to host memory and SSD.

  • Project
    microsoft/Tutel

    Tutel is Microsoft’s open‑source, high‑performance MoE library for large language models. It provides dynamic parallelism, low‑precision inference (NVFP4, MXFP4, FP8, BF16), support for up to 1 M‑token contexts, vision extensions, and Docker images that serve models like DeepSeek‑V3.2, Kimi‑K3, GLM‑5.x, Qwen‑3, and GPT‑OSS on both NVIDIA and AMD GPUs. Install via pip from the Git repo or build from source, then run the supplied Docker containers or use the Python API for custom MoE routing.

  • Project
    Luce-Org/lucebox

    A speculative inference engine optimized for heterogeneous consumer GPUs and APUs, providing high-throughput local LLM execution via custom kernels and hardware-tuned optimizations.