pegainfer-project/pegainfer

What it solves

PegaInfer is a high-performance LLM inference engine designed to eliminate the overhead associated with heavy model framework runtimes like PyTorch or ONNX. By removing these dependencies, it achieves significantly faster cold start times and a much smaller memory footprint compared to traditional inference frameworks.

How it works

The engine is built entirely in Rust and CUDA, featuring hand-written kernels and schedulers. It utilizes a GPU-first runtime where model execution stays in native paths. Key technical implementations include:

• Custom Kernels: Uses CUDA for decode-critical paths, Triton AOT for specific model compatibility, FlashInfer for paged attention and sampling, and cuBLAS for matrix multiplication.
• CUDA Graphs: Employed on Qwen decode paths to eliminate kernel launch overhead.
• KV Management: Implements a KV cache and a host-tier restore system (pegaflow) that allows evicted prefixes to be restored from host DRAM instead of being recomputed.
• Multi-GPU Support: Integrates NCCL for reductions and a DeepEP shim for specific model architectures.

Who it’s for

Developers and engineers who need a production-grade, lightweight inference server with minimal startup latency and low resident memory usage, particularly those serving frontier-scale models like Qwen and Kimi.

Highlights

• Zero-Framework Dependency: No PyTorch or ONNX required at runtime.
• Rapid Startup: Cold start to HTTP-ready in ~3 seconds, compared to ~70 seconds for vLLM.
• Low Footprint: Approximately 5” smaller resident memory footprint than vLLM.
• Broad Model Support: Supports Qwen3, Qwen3.5, DeepSeek-V2-Lite, Kimi-K2, and GLM-5.2.
• OpenAI Compatible: Provides /v1/completions and /v1/chat/completions endpoints.