Neroued/ninfer

High-performance single-GPU inference for selected model checkpoints and GPUs.

What it solves

NInfer is a high-performance inference engine designed specifically for the NVIDIA GeForce RTX 5090 to run Qwen3.5 Dense and MoE architectures. It aims to maximize single-GPU inference performance for text, image, and video prompts through a specialized runtime that minimizes overhead.

How it works

Built from scratch in C++/CUDA, the engine is optimized for the sm_120a architecture. It uses a fixed number of execution lanes (1-8) and a single resident model. The system employs a shared KV pool for active requests and retained prefixes, allowing for efficient resource-aware long-context reuse. It supports various weight formats (BF16, INT8, FP8, NVFP4) and speculative decoding techniques like MTP and DFlash to accelerate token generation.

Who it’s for

Developers and researchers who own an RTX 5090 and want to maximize the throughput and latency of Qwen3.5 models locally via a CLI or OpenAI-/Anthropic-compatible HTTP APIs.

Highlights

  • RTX 5090 Specialization: Specifically tuned for the Blackwell architecture to achieve maximum single-GPU performance.
  • Multimodal Support: Handles text, image, multi-image, and video prompts.
  • Advanced Decoding: Implements MTP speculative decoding and DFlash for faster generation.
  • Context Management: Features private and shared exact-prefix reuse with Device/Host state retention to reduce TTFT (Time to First Token).
  • High Throughput: Supports chunked prefill and exact-batch CUDA Graph decode.
  • API Compatibility: Provides OpenAI and Anthropic compatible endpoints for easy integration.

Related

  • Project
    omnimind-ai/OmniInfer

    OmniInfer is a cross‑platform inference engine that lets you run LLMs and vision‑language models locally via a simple CLI and OpenAI‑compatible HTTP server. It supports many backends (llama.cpp, MLX, MNN, etc.), works on desktop, Android and iOS, and provides hardware‑aware optimisation and a terminal UI for chatting.

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Dispatch
    Qwen2.5-1M Release: Open‑Source 7B and 14B Models with 1M‑Token Context and vLLM‑Based Inference Framework

    Qwen releases open‑source Qwen2.5‑7B‑Instruct‑1M and Qwen2.5‑14B‑Instruct‑1M models that support up to 1 million token contexts, accompanied by an optimized vLLM‑based inference framework that delivers 3×–7× faster prefill and retains short‑task performance comparable to GPT‑4o‑mini.

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a GPU‑kernel library (with a Python package) that provides ultra‑fast, low‑precision attention, GEMM, MoE, and sampling operations for LLM inference. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM) for a wide range of NVIDIA GPUs, from Turing up to Blackwell, and is already used by popular inference servers like vLLM and TensorRT‑LLM.

  • Dispatch
    Qwen 3.5‑397B‑A17B release: hybrid linear‑attention MoE model with 1 M token context and state‑of‑the‑art multimodal performance

    Qwen 3.5‑397B‑A17B is a 397 billion‑parameter multimodal model that activates only 17 billion parameters per token, delivering state‑of‑the‑art performance on language, coding, reasoning and vision tasks while being up to 19× faster than its predecessor.