weicj/vLLM-2080Ti-Definitive

The definitive vLLM runtime for dual RTX 2080 Ti 22GB + NVLink, delivering Qwen 27B local inference with maximum 100+ tok/s single-request decode with support of FP8 weight ( Join Discord :https://discord.gg/VFqVVySdMS )

What it solves

This project provides a specialized vLLM runtime designed specifically for dual NVIDIA RTX 2080 Ti GPUs (SM75) with 22GB memory mods. It allows users to run large language models (LLMs) like Qwen3.x 27B/35B and Gemma4 31B on older Turing-architecture hardware, transforming these GPUs into a high-performance inference platform for personal-agent style workloads.

How it works

It is a hardware-focused fork of vLLM that integrates several acceleration kernels and patches specifically tuned for the SM75 architecture. It utilizes Marlin, FlashQLA/FlashInfer, TurboQuant/INT8 KV, and MTP (Multi-Token Prediction) to optimize throughput and context window size. The project includes a set of pre-validated "profiles" (e.g., safe, normal, fast, aggressive) that configure the runtime for specific model weights and KV cache precision to balance quality and performance.

Who it’s for

AI enthusiasts and developers using dual RTX 2080 Ti 22GB GPUs with NVLink or PCIe P2P, who want to run 27B-35B class models with large context windows (up to 256K) on a budget-friendly, secondary-market hardware setup.

Highlights

  • Hardware Optimization: Specifically tuned for dual RTX 2080 Ti 22GB (TP=2) with NVLink support.
  • High Throughput: Validated Qwen3.6 27B FP8 route reaching 100+ tokens per second for single-request decode.
  • Extensive Context Support: Supports native 256K context for Qwen3.x 27B and 136K-178K for Qwen3.x 35B.
  • Integrated Tooling: Includes a build.sh script for automated environment setup and a launcher.sh interactive service manager for easy profile application and server management.
  • Broad Model Support: Validated routes for Qwen3.x (27B/35B) and experimental support for Gemma4 31B.

Related

  • Project
    syv-ai/qwen38-27b-rtx3090

    An optimized serving setup for Qwen3.8-27B on a single RTX 3090, using speculative decoding and quantization to achieve high throughput and low latency.

  • Project
    1CatAI/1Cat-vLLM

    A vLLM fork optimized specifically for Tesla V100 (SM70) GPUs, enabling efficient serving of modern Qwen-class AWQ and FP8 models with long-context support.

  • Project
    noonghunna/club-3090

    A collection of validated Docker Compose recipes and tooling for serving LLMs locally on RTX 3090 GPUs, optimizing for throughput and VRAM management.

  • Project
    turboderp-org/exllamav3

    An inference library for running local LLMs on consumer GPUs, featuring the efficient EXL3 quantization format and support for a wide range of model architectures.

  • Dispatch
    vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard

    vLLM has achieved top rankings on the Artificial Analysis leaderboard for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B by implementing aggressive kernel fusion and speculative decoding optimizations.