local-inference-lab/rtx6kpro

RTX PRO 6000 Blackwell LLM Wiki

What it is – A community‑maintained, highly detailed wiki that documents how to run state‑of‑the‑art large language models on NVIDIA RTX PRO 6000 Blackwell (SM120) PCIe GPUs. It bundles reproducible Docker images, exact launch commands, benchmark tables, quantisation‑fidelity checks, and low‑level hardware‑tuning notes.

Why it matters – Running frontier models (GLM‑5.x, Qwen 3.8‑Flash, DeepSeek‑V4, Kimi, Xiaomi MiMo, etc.) at high throughput on a PCIe‑only Blackwell server is non‑trivial. The repo captures the full stack – from Docker build scripts to PCIe topology advice – so users can reproduce published speed numbers and avoid common pitfalls.

Key contents

• Unified Docker launch guide (docs/unified‑vllm‑docker.md) that builds a single image supporting vLLM, SGLang, speculative decoding, and optional LMCache.
• Model‑specific runbooks (e.g., models/glm-5.3-flash.md, models/qwen38‑flash‑next.md) with exact vllm/sglang command lines, TP/MTP/DFlash settings, and measured throughput.
• Benchmark & quality sections – consolidated throughput tables, Kullback‑Leibler‑Divergence (KLD) quantisation fidelity checks, and task‑level quality comparisons (e.g., Czech tutoring workload).
• Hardware & topology docs – SM120 vs SM100 architecture, PCIe‑5.0 bandwidth, power‑limit sweeps, and chassis‑specific wiring guides.
• Optimization notes – speculative decoding, NVFP4 quantisation, B12X FP8 kernels, NCCL tuning, PCIe‑oneshot all‑reduce, I/O tuning.
• Troubleshooting & onboarding – common issues, newcomer onboarding guide, and an acronym glossary to keep the dense terminology readable.

Who should use it

• Engineers and researchers who have access to RTX PRO 6000 Blackwell servers and need reproducible, high‑performance LLM serving.
• Teams building internal inference services that require precise speed/quality reporting.
• Anyone looking to understand low‑level GPU‑specific optimisations (MTP, DSpark, DFlash, NVFP4, etc.) for frontier models.

Getting started

1. Read the onboarding doc (docs/newcomer‑onboarding.md) to understand the expected workflow.
2. Build the unified Docker image following the guide in docs/unified‑vllm‑docker.md.
3. Pick a model hub (e.g., models/glm-5.3‑flash.md) and copy the provided launch command.
4. Verify your hardware matches the documented configuration (GPU count, PCIe topology, power limits).
5. Run the benchmark or quality scripts linked from the model page to confirm you reproduce the reported numbers.

Bottom line – This repository is a living, reproducible reference for anyone who wants to squeeze the maximum inference performance out of NVIDIA’s newest Blackwell GPUs while keeping rigorous quality and benchmarking standards.