Neroued/ninfer

High-performance single-GPU inference for selected model checkpoints and GPUs.

What it solves

NInfer is a high-performance inference engine designed specifically for a closed set of Qwen model checkpoints on a single NVIDIA GeForce RTX 5090. It eliminates the overhead of general-purpose runtimes to achieve maximum single-GPU performance for text, image, and video prompts.

How it works

Built from scratch in C++/CUDA, NInfer uses specialized artifacts (.ninfer files) rather than standard Transformers checkpoints. It leverages hardware-specific optimizations like W4A4 Tensor Core MMA for prefill and A16 NVFP4 kernels for decode in its NVFP4 profile. The engine supports chunked prefill, CUDA Graph decode, and speculative decoding (MTP and DFlash) to accelerate generation.

Who it’s for

Developers and researchers with an NVIDIA RTX 5090 who need the absolute highest possible throughput and lowest latency for Qwen 3.6 and 3.8 models locally.

Highlights

  • Hardware Optimized: Specifically tuned for the RTX 5090 (sm_120a) and CUDA 13.1.
  • Multimodal Support: Handles text, multi-image, and video inputs.
  • Speculative Decoding: Implements MTP (Multi-Token Prediction) and DFlash (for 35B-A3B target) to increase decode speed.
  • API Compatible: Provides a local CLI and HTTP servers compatible with OpenAI and Anthropic message formats.
  • High Throughput: Capable of reaching over 1,300 aggregate decode tokens per second on specific model profiles.
  • Memory Management: Features BF16 and INT8 group-64 KV cache with configurable capacity.

Related

  • Project
    InfiniTensor/InfiniTensor

    A high-performance inference engine for GPUs and AI accelerators that enables efficient model deployment and academic validation.

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Dispatch
    Running Qwen 3.6 27B Q8 at 80+ Tok/s with RTX 5080 and RTX 3090

    A technical guide on achieving 80+ tokens per second on Qwen 3.6 27B Q8 using a dual-GPU setup consisting of an RTX 5080 and a refurbished RTX 3090.

  • Project
    flashinfer-ai/flashinfer

    A high-performance GPU kernel library and generator for LLM inference that optimizes attention, GEMM, and MoE operations across diverse NVIDIA GPU architectures.

  • Project
    syv-ai/qwen38-27b-rtx3090

    An optimized serving setup for Qwen3.8-27B on a single RTX 3090, using speculative decoding and quantization to achieve high throughput and low latency.