syv-ai/qwen38-27b-rtx3090

Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks

What it solves

This project provides a highly optimized serving setup for the Qwen3.8-27B model on a single consumer-grade NVIDIA RTX 3090 (24 GB VRAM). It addresses the challenge of running a large model with high throughput and low latency on limited hardware by employing advanced quantization, speculative decoding, and memory management techniques.

How it works

The system uses vLLM as its base and applies several layers of optimization:

  • Quantization: It requantizes embedding matrices and the LM head to int8/int4 to free up VRAM and increase speed.
  • Speculative Decoding: It offers multiple speculation modes, including MTP (Multi-Token Prediction) and DFlash2, which propose multiple tokens per step to accelerate generation.
  • Context Management: It implements prefix caching to avoid re-processing documents and supports extended context windows (up to 240k tokens) via the KVarN 4/2-bit KV cache.
  • Operational Modes: It provides two distinct configurations: a "batch" mode for high-throughput API backends and a "single-user" mode optimized for low-latency chatting.

Who it’s for

Developers and AI researchers who own a single RTX 3090 and want to run Qwen3.8-27B with production-level performance, high context windows, and an OpenAI-compatible API.

Highlights

  • High Throughput: Achieves up to ~1,035 tokens per second in batch mode.
  • Low Latency: Reaches over 120 tokens per second in single-user mode using DFlash2 speculation.
  • Massive Context: Supports up to 240k tokens of context with the KVarN integration.
  • Lossless Speculation: Speculative decoding is exact, maintaining the original model's distribution.
  • OpenAI Compatible: Provides an API with optional key authentication.

Related

  • Project
    noonghunna/club-3090

    A collection of validated Docker Compose recipes and tooling for serving LLMs locally on RTX 3090 GPUs, optimizing for throughput and VRAM management.

  • Project
    weicj/vLLM-2080Ti-Definitive

    A specialized vLLM fork optimized for dual RTX 2080 Ti 22GB GPUs, enabling high-performance inference for 27B-35B class models on Turing-architecture hardware.

  • Dispatch
    Running Qwen 3.6 27B Q8 at 80+ Tok/s with RTX 5080 and RTX 3090

    A technical guide on achieving 80+ tokens per second on Qwen 3.6 27B Q8 using a dual-GPU setup consisting of an RTX 5080 and a refurbished RTX 3090.

  • Project
    MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark

    Optimized Docker scripts for serving Qwen3.8-27B on NVIDIA DGX Spark using SGLang, featuring pre-tuned configurations for MTP, DSpark, and DFlash2 speculative decoding.

  • Project
    Neroued/ninfer

    A C++/CUDA inference engine optimized for maximum performance of Qwen checkpoints on a single NVIDIA RTX 5090, supporting multimodal inputs and speculative decoding.