openinfer-project/openinfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

What it solves

openinfer is a high-performance LLM inference engine designed to eliminate the overhead associated with heavy model framework runtimes like PyTorch or ONNX. By removing these dependencies, it achieves significantly faster cold start times, a smaller memory footprint, and reduced latency for multi-turn chat and agentic workflows.

How it works

Built entirely in Rust and CUDA, the engine uses hand-written kernels and a custom scheduler. It leverages CUDA Graphs to eliminate kernel launch overhead on decode paths and employs a tiered KV cache system (via openinfer-kv-offload) that can restore evicted prefixes from host DRAM instead of recomputing them. For specific models, it integrates Triton AOT kernels, FlashInfer for paged attention, and NCCL for multi-GPU communication.

Who it’s for

Developers and engineers building production-grade LLM services who require minimal startup latency, low resident memory usage, and high throughput for frontier-scale models without the weight of a Python-based runtime.

Highlights

  • Zero-Framework Runtime: No PyTorch or ONNX; pure Rust and CUDA for the default build.
  • Rapid Startup: Cold starts in approximately 3 seconds, compared to significantly longer times in frameworks like vLLM.
  • Low Memory Footprint: Approximately 5x smaller resident memory usage when idle compared to vLLM.
  • Frontier Model Support: Supports models including Qwen3, Qwen3.5, DeepSeek-V2-Lite, and the trillion-parameter Kimi-K2.
  • OpenAI Compatible: Provides a /v1/completions endpoint for easy integration.