wtdcode/vllm-backport

What it solves

This project provides a specialized fork of the vLLM inference engine designed to run large, modern frontier models on older NVIDIA GPU architectures (specifically Ampere-based cards like the A6000, A100, and RTX 30-series). It addresses hardware limitations that often prevent these cards from running the latest high-parameter or complex-architecture models efficiently.

How it works

It functions as a modified version of vLLM that integrates specific optimizations, such as support for native FP4 quantization, expert parallelism, and custom CUDA graph configurations. It includes a paired fork of LMCache for efficient KV-cache management and provides pre-built Docker images tailored to specific GPU architectures (sm80, sm86, sm89). The project allows users to bypass standard vLLM constraints by enabling features like CPU offloading for embeddings and specific memory-saving configurations for MoE (Mixture-of-Experts) models.

Who it’s for

This is for infrastructure engineers and AI researchers who need to serve high-end models like DeepSeek-V4 or Qwen3.8 on existing enterprise or high-end consumer hardware that lacks the latest Hopper-generation (H100/H200) capabilities.

Highlights

• Hardware Compatibility: Optimized for Ampere (A100, A6000, RTX 30xx) and Ada (RTX 4090, L40S) architectures.
• Advanced Quantization: Native support for FP4 and FP8, along with AWQ W4A16 for memory-efficient model serving.
• Integrated Caching: Built-in support for LMCache to manage KV-cache across distributed environments.
• Performance Tuning: Includes specialized configurations for speculative decoding (MTP, DSpark) and CUDA graph capture to maximize throughput on older hardware.