local-inference-lab/rtx6kpro

RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink

What it solves

This repository provides a comprehensive technical guide and "field wiki" for deploying and optimizing frontier Large Language Models (LLMs) on NVIDIA RTX PRO 6000 Blackwell (SM120) PCIe systems. It eliminates the need for "tribal knowledge" by providing reproducible Docker builds, exact runbooks, and hardware-specific tuning for high-performance inference without NVLink.

How it works

The project organizes detailed documentation and scripts around several core pillars:

  • Model-Specific Runbooks: Step-by-step instructions for running models like GLM, DeepSeek, Kimi, and Qwen, including specific Tensor Parallelism (TP) and Decode Context Parallelism (DCP) settings.
  • Optimization Guides: Technical deep-dives into speculative decoding (MTP, DSpark, DFlash), quantization formats (NVFP4, MXFP8), and kernel optimizations (B12X, FlashInfer).
  • Hardware Tuning: Guidance on PCIe topology, bandwidth optimization, and power limits for multi-GPU systems (4, 8, or 16 GPUs).
  • Validation Tools: Methodology for using Kullback-Leibler Divergence (KLD) to check quantization fidelity and behavioral fidelity tests to ensure model quality.

Who it’s for

It is designed for AI engineers and researchers operating NVIDIA Blackwell-based PCIe hardware who need to maximize throughput and quality for frontier LLMs using engines like vLLM and SGLang.

Highlights

  • Reproducible Environments: Includes Docker images and build scripts to ensure consistent runtime environments.
  • Advanced Quantization Support: Detailed implementation notes for NVFP4 and MXFP8 floating-point formats.
  • Hardware-Specific Benchmarks: Throughput and quality tables specifically for RTX PRO 6000 Blackwell systems.
  • Topology Optimization: Specific advice for PCIe-switch systems and NCCL tuning to overcome the lack of NVLink.

Related

  • Project
    kacper-daftcode/vLLM-Moet

    A patched vLLM runtime for NVIDIA Blackwell GPUs that uses 2-bit quantization and tiered NVMe offloading to serve massive MoE models like GLM-5.2 and DeepSeek-V4-Flash on consumer hardware.

  • Project
    noonghunna/club-3090

    A collection of validated Docker Compose recipes and tooling for serving LLMs locally on RTX 3090 GPUs, optimizing for throughput and context window stability.

  • Project
    NVIDIA/TensorRT-LLM

    An open-source optimization library from NVIDIA that maximizes inference performance, throughput, and efficiency for LLMs and visual generation models on NVIDIA GPUs.

  • Project
    ztxz16/fastllm

    A high-performance C++ LLM inference engine that enables hybrid deployment of dense and MoE models across GPU, CPU, NUMA, and disk to maximize hardware utilization.

  • Project
    cfregly/ai-performance-engineering

    A comprehensive collection of code and resources for optimizing AI systems, focusing on GPU tuning, distributed training, and high-throughput inference scaling.