1CatAI/1Cat-vLLM

V100 / SM70-focused vLLM engineering fork for modern LLM inference.

1Cat‑LLM (formerly 1Cat‑vLLM)

What it is – A fork of the open‑source vLLM inference engine that is engineered specifically for NVIDIA Volta GPUs (SM70, e.g. Tesla V100). It adds kernels, data‑type handling and scheduling tricks that make large language models run much faster on this older hardware.

Key capabilities

  • Volta‑first optimisation – custom FlashAttention kernels, FP8/E4M3 KV‑cache support, CUDA‑graph scheduling and mixed‑precision paths that target the V100’s Tensor Cores.
  • Model support – works with the GGUF checkpoint format and a set of community‑quantised Qwen 3.x, DeepSeek‑V4, GLM‑5.3 and other 27‑30 B‑parameter models. Example checkpoints are listed (e.g. QUASAR‑QAT/Qwen3.8‑27B‑NVFP4).
  • Performance – on a 4‑GPU V100‑SXM2 (32 GB) system the library reports up to ~260 tokens / s for a 27 B model with the DFlash2 speculative‑decoding engine; typical decode rates for 128 K‑token contexts are ~50 tok/s.
  • Quality‑aware benchmarking – the README pairs raw throughput numbers with dataset scores (MBPP, HumanEval, GSM8K) and speculative‑decoding acceptance rates, so users can see the trade‑off between speed and output quality.
  • GGUF & mixed‑load – recent releases add support for the GGUF model format and the ability to load parts of a model in different precisions, expanding the range of models that fit on V100 memory.

Who it helps – Researchers or engineers who must run large LLMs on legacy Volta clusters (e.g., in academic HPC centres or cost‑constrained cloud setups). By extracting extra TFLOP/s from V100s, 1Cat‑LLM lets them serve or evaluate 27‑30 B‑parameter models without buying newer A‑series GPUs.

How to use it

  1. Install the Python package (the same name as the old 1Cat‑vLLM wheel) and the matching native extension compiled for your V100 driver.
  2. Choose a supported checkpoint (GGUF or the listed Qwen/DeepSeek variants) and launch inference via the familiar vLLM API, optionally enabling the DFlash2 speculative decoder.
  3. Tune KV‑cache dtype (fp8_e4m3 or e5m2) and other launch flags as described in the design docs for best throughput on your hardware.

Limitations – The speed figures are tied to very specific hardware (four V100‑SXM2 GPUs) and benchmark contracts; results will differ on other setups. Some features (GGUF support, mixed‑load) require building from source rather than using the pre‑built wheel.

Status – Actively maintained as of October 2026 (v1.5.1). The project is transitioning to the new name 1Cat‑LLM while keeping existing release artifacts for compatibility.

Related

  • Project
    weicj/vLLM-2080Ti-Definitive

    A specialized vLLM fork optimized for NVIDIA Turing (SM75) GPUs, enabling high-performance inference for 27B-35B parameter models on hardware like the RTX 2080 Ti.

  • Dispatch
    vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard

    vLLM has achieved top rankings on the Artificial Analysis leaderboard for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B by implementing aggressive kernel fusion and speculative decoding optimizations.

  • Project
    snowflakedb/ArcticInference

    An open-source vLLM plugin that optimizes LLM and embedding inference to increase throughput and reduce latency using advanced parallelism and speculative decoding.

  • Dispatch
    vLLM DeepSeek V4 Support: Efficient Long-context Attention

    vLLM now supports DeepSeek V4-Pro and V4-Flash, implementing a new attention mechanism that enables context lengths up to one million tokens with significant KV cache memory savings.

  • Dispatch
    vLLM TML Inkling Support

    vLLM has announced Day-0 support for TML Inkling, a 1T-parameter multimodal model, delivering up to 380 tok/s/user on 4 GB200 GPUs through specialized architectural optimizations.