guoqingbao/xinfer

Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.

What it solves

xInfer is a high-performance LLM inference engine designed to eliminate the overhead of Python and PyTorch. It provides a portable, production-ready environment for running large models with minimal memory footprints and high throughput, specifically targeting the ability to run large MoE (Mixture-of-Experts) models on consumer-grade GPUs.

How it works

Built entirely in Rust, xInfer implements a native backend that bypasses the Python runtime. It utilizes advanced optimization techniques including native Flash Attention, FlashInfer, CUDA Graphs, and continuous batching. To handle memory constraints, it uses "TurboQuant" (2–4 bit KV cache compression) and supports various quantization formats like NVFP4, FP8, and GGUF. It also supports Prefill-Decode Disaggregation, allowing the prompt processing (prefill) and token generation (decode) to be split across different GPUs or machines to prevent stalls.

Who it’s for

It is intended for developers and operators who need a fast, lightweight, and cross-platform (Linux, Windows, macOS) inference server that is compatible with OpenAI and Anthropic APIs, as well as those wanting to deploy large models on limited hardware.

Highlights

  • Zero Python Dependencies: Pure Rust backend for improved portability and performance.
  • Aggressive KV Compression: TurboQuant extends context length up to 4.3x, enabling 30B+ models to run on single 24/32 GB GPUs.
  • Broad Model Support: Compatible with Llama, Qwen, Mistral, GLM, DeepSeek, Phi, and Gemma, including multimodal versions.
  • Production Features: Includes a built-in ChatGPT-style Web UI, MCP tool calling, and structured outputs via guided decoding.
  • Multi-Node Scaling: Supports tensor parallelism across multiple machines using TCP-based NCCL bootstrap without requiring MPI.

Related

  • Project
    pegainfer-project/pegainfer

    A pure Rust and CUDA LLM inference engine that removes framework runtimes like PyTorch to provide faster startup times and a smaller memory footprint.

  • Project
    lablup/mlxcel

    A high-performance Rust-based inference runtime and server for LLMs and VLMs on Apple Silicon and NVIDIA CUDA, removing the need for Python environments during deployment.

  • Project
    Michael-A-Kuykendall/shimmy

    A lightweight, pure-Rust OpenAI-compatible inference server for GGUF models that uses WebGPU for cross-platform GPU acceleration.

  • Project
    Tiiny-AI/PowerInfer

    PowerInfer is a CPU/GPU hybrid LLM inference engine that uses activation locality to run large models quickly on consumer-grade GPUs by offloading 'cold' neurons to the CPU.

  • Project
    flashinfer-ai/flashinfer

    A high-performance GPU kernel library and generator for LLM inference that optimizes attention, GEMM, and MoE operations across diverse NVIDIA GPU architectures.