1CatAI/1Cat-vLLM

V100 / SM70-focused vLLM engineering fork for modern LLM inference.

What it solves

1Cat-vLLM is a specialized fork of vLLM designed to prevent Tesla V100 (SM70) GPUs from becoming obsolete in the era of modern Large Language Models. It optimizes the serving of modern models, specifically Qwen-class AWQ and experimental FP8 models, on older Volta architecture hardware where standard vLLM might not be as efficient or compatible.

How it works

The project integrates several hardware-specific optimizations to enable high-performance inference on V100s:

  • Custom Kernels: It incorporates TurboMind-derived SM70 kernels and a dedicated FLASH_ATTN_V100 backend for both decode and prefill operations.
  • AWQ Support: It provides optimized 4-bit AWQ inference paths for dense and MoE Qwen models on SM70.
  • Memory & Context Management: It includes runtime defaults optimized for long-context serving (up to 256K context) and manages memory budgets for multimodal inputs.
  • Speculative Decoding: It supports MTP (Multi-Token Prediction) speculative decoding as an optional path for specific workloads.
  • Experimental Paths: It includes research-level support for FP8 models, FP8 KV-caches, and DFlash.

Who it’s for

This project is for individual developers, studios, and teams who continue to use Tesla V100 GPUs and want to deploy modern, high-performance LLMs with OpenAI-compatible APIs.

Highlights

  • V100-First Optimization: Specifically tuned for SM70 architecture rather than being a general-purpose fork.
  • Long-Context Capability: Public profiles default to 256K context where memory allows.
  • Multimodal Ready: Supports image inputs by default on the SM70 path.
  • OpenAI Compatibility: Validated with common OpenAI-style clients like Cherry Studio and OpenClaw.
  • Prebuilt Wheels: Provides easy installation via prebuilt wheels for Ubuntu 24.04 and CUDA 12.8.

Related

  • Project
    weicj/vLLM-2080Ti-Definitive

    A specialized vLLM fork optimized for dual RTX 2080 Ti 22GB GPUs, enabling high-performance inference for 27B-35B class models on Turing-architecture hardware.

  • Project
    xLLM-AI/xllm

    xLLM is an efficient LLM inference framework optimized for Chinese AI accelerators, designed for high-throughput, low-latency enterprise-grade deployment.

  • Dispatch
    vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard

    vLLM has achieved top rankings on the Artificial Analysis leaderboard for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B by implementing aggressive kernel fusion and speculative decoding optimizations.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.

  • Dispatch
    vLLM DeepSeek V4 Support: Efficient Long-context Attention

    vLLM now supports DeepSeek V4-Pro and V4-Flash, implementing a new attention mechanism that enables context lengths up to one million tokens with significant KV cache memory savings.