pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

What it solves

PegaInfer is a high-performance LLM inference server designed for production environments. It eliminates the need for a Python runtime or PyTorch, reducing memory overhead and startup times while providing an OpenAI-compatible API for serving various large language models.

How it works

Built entirely in Rust and CUDA, the project uses a modular architecture where shared infrastructure (GPU primitives, sampling, and KV cache) is decoupled from model-specific logic. Each supported model owns its own scheduler, state management, and kernels, allowing for optimized execution paths tailored to specific model architectures (such as dense, MoE, or hybrid-attention models).

Who it’s for

It is intended for developers and engineers who need to deploy LLMs with minimal latency, low memory footprint, and high throughput on NVIDIA GPUs, particularly those seeking an alternative to Python-based inference stacks.

Highlights

  • Zero Python Runtime: Operates without PyTorch or Python at runtime, significantly reducing resident memory and cold-start times.
  • OpenAI Compatible: Provides a standard /v1/completions and /v1/chat/completions API with streaming support.
  • Broad Model Support: Supports a variety of model families including Qwen3, Qwen3.5, Gemma 4, DeepSeek-V2-Lite, Kimi-K2/K3, and GLM-5.2.
  • Advanced Inference Techniques: Implements tensor parallelism, prefix caching, KV offloading, and speculative decoding (DFlash/DSpark).
  • Hardware Optimized: Specifically tuned for NVIDIA GPUs, including support for Blackwell architecture and specialized kernels.

Related

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Project
    Neroued/ninfer

    A C++/CUDA inference engine optimized for the NVIDIA RTX 5090, providing maximum performance for Qwen3.5 Dense and MoE models across text, image, and video.

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a GPU‑kernel library (with a Python package) that provides ultra‑fast, low‑precision attention, GEMM, MoE, and sampling operations for LLM inference. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM) for a wide range of NVIDIA GPUs, from Turing up to Blackwell, and is already used by popular inference servers like vLLM and TensorRT‑LLM.

  • Project
    omnimind-ai/OmniInfer

    OmniInfer is a cross‑platform inference engine that lets you run LLMs and vision‑language models locally via a simple CLI and OpenAI‑compatible HTTP server. It supports many backends (llama.cpp, MLX, MNN, etc.), works on desktop, Android and iOS, and provides hardware‑aware optimisation and a terminal UI for chatting.

  • Project
    tracel-ai/burn

    A deep learning framework and tensor library in Rust that unifies training and inference into a single codebase for seamless deployment across diverse hardware.