ztxz16/fastllm

fastllm是后端无依赖的高性能大模型推理库。同时支持张量并行推理稠密模型和混合模式推理MOE模型，任意10G以上显卡即可推理满血DeepSeek。双路9004/9005服务器+单显卡部署DeepSeek满血满精度原版模型，单并发20tps；INT4量化模型单并发30tps，多并发可达60+。

What it solves

fastllm is a high-performance LLM inference library designed to run large models (like DeepSeek R1 671B) on hardware with limited VRAM. It removes the dependency on PyTorch by using custom C++ operators, allowing for efficient execution across a wide range of GPUs and CPUs, including older hardware and specialized NPUs.

How it works

The library implements its own low-level operators in C++ and supports several optimization techniques:

  • Hybrid Inference: It can split model execution between GPU and CPU (or multiple NUMA nodes), enabling the deployment of massive MoE (Mixture of Experts) models on a single GPU by offloading MoE layers to the CPU.
  • Flexible Quantization: It supports FP8, INT8, INT4, and AWQ formats, with the ability to perform online quantization or export models to specific types for faster loading.
  • Tensor Parallelism: It supports multi-GPU tensor parallel inference, including odd numbers of cards (e.g., 3, 5, 7).
  • Broad Hardware Support: It is compatible with NVIDIA (M40 to 5090), AMD (MI50, 7900), and various domestic Chinese NPUs (Tianshu, Muxi, Ascend).

Who it’s for

  • Developers and researchers who want to run massive models on consumer-grade or older hardware.
  • Users needing a lightweight, PyTorch-free inference engine for deployment on Linux, Windows, or Android.
  • Those looking for an easy-to-use CLI tool to start LLM servers, WebUIs, or local chats.

Highlights

  • Low VRAM Requirement: Can run the full DeepSeek R1 671B model on a single card if VRAM > 10GB (via CPU offloading).
  • PyTorch-Free: Custom C++ implementation for better performance and easier portability.
  • Diverse Quantization: Supports FP8 inference on any GPU and dynamic quantization.
  • Versatile Deployment: Includes an OpenAI-compatible API server, a WebUI, and a TUI deployment wizard.
  • Cross-Platform: Supports NVIDIA, AMD, and various NPUs, and can be compiled for Android.

Related

  • Project
    OpenNMT/CTranslate2

    A C++ and Python library for efficient Transformer model inference, using quantization and custom runtimes to accelerate execution and reduce memory usage on CPU and GPU.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.

  • Project
    guqiong96/Lvllm

    An extension of vLLM that enables hybrid CPU-GPU inference for large MoE models by utilizing both VRAM and system memory through a NUMA-aware architecture.

  • Project
    leejet/stable-diffusion.cpp

    A pure C/C++ implementation for diffusion model inference, enabling lightweight and efficient image and video generation across multiple platforms and hardware backends.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.