weicj/vLLM-2080Ti-Definitive

The definitive vLLM runtime for dual RTX 2080 Ti 22GB + NVLink, delivering Qwen 27B local inference with maximum 200+ tok/s single-request decode with support of FP8 weight ( Join Discord :https://discord.gg/VFqVVySdMS )

What it solves

This project provides a specialized vLLM inference runtime optimized specifically for NVIDIA Turing architecture GPUs (SM75), such as the RTX 2080 Ti, Tesla T4, and Quadro RTX series. It enables high-performance serving of larger models (like 27B and 35B parameter models) on older hardware by leveraging specific hardware properties and optimized kernels.

How it works

It is a hardware-focused fork of vLLM that integrates several acceleration components and SM75-specific source changes. It uses Marlin, FlashInfer/FlashQLA, TurboQuant/INT8 KV, and CUDA Graph support to maximize throughput. The project includes a dedicated launcher and pre-configured profiles to manage Tensor Parallelism (TP) and Pipeline Parallelism (PP) across multiple GPUs (e.g., 2x RTX 2080 Ti with NVLink).

Who it’s for

Users with NVIDIA Turing-generation GPUs (SM75) who want to run large language models locally with high throughput and efficiency, particularly those using dual RTX 2080 Ti or Tesla T10/T4 setups.

Highlights

  • SM75 Optimization: Specifically tuned for Turing GPUs, allowing 27B-35B class models to be served efficiently.
  • High Throughput: Achieves significant decode speeds (e.g., over 200 tok/s for Qwen 27B NVFP4 on 2x 2080 Ti).
  • Integrated Launcher: An interactive launcher.sh script for easy configuration of GPU topology, model paths, and performance modes (normal, fast, aggressive, safe).
  • ** uma Hardware Support**: Validated routes for 2x RTX 2080 Ti, 2x Tesla T10, and 4x Tesla T10 configurations.
  • Broad Weight Support: Tested with various weight formats including FP8, NVFP4, and INT4 (W4A16).

Related

  • Project
    1CatAI/1Cat-vLLM

    1Cat‑LLM is a vLLM fork that optimises LLM inference for NVIDIA Volta (Tesla V100) GPUs, offering custom kernels, FP8/E4M3 KV‑cache, GGUF support and speculative decoding to achieve up to ~260 tokens / s on 27‑B models while reporting quality‑aware benchmarks.

  • Project
    kacper-daftcode/vLLM-Moet

    vLLM-Moet provides optimized vLLM images and patches to run frontier Mixture-of-Experts models on Blackwell (SM120) consumer GPUs by implementing custom kernels, efficient KV caching, and 2-bit expert quantization.

  • Project
    MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks

    A deployment framework for running Qwen3.8-Flash-Next models across dual-DGX Spark nodes using vLLM, featuring automated kernel patching, multi-node synchronization, and optimized KV cache management.

  • Dispatch
    vLLM Qwen3.5 Performance Optimization

    vLLM has achieved over 25,000 total tokens per second (TPS) per GPU for Qwen3.5 on GB200 NVL72 systems through Blackwell-optimized kernels, hybrid cache state transfer, and async scheduling.

  • Project
    localai-org/vllm.cpp

    A high-performance C++20 inference engine that mirrors vLLM's features and throughput but removes the Python/PyTorch dependency for a 140x smaller deployment footprint.