Running Qwen 3.6 27B Q8 at 80+ Tok/s with RTX 5080 and RTX 3090

A dual-GPU configuration combining an NVIDIA RTX 5080 and an RTX 3090 can achieve inference speeds of 80 to 90+ tokens per second (tok/s) when running the Qwen 3.6 27B Q8 model. This performance is made possible through a combination of specific BIOS settings, the nvidia-open driver, and llama.cpp optimizations including Multi-Token Prediction (MTP) and speculative decoding.

Hardware Configuration

To support two high-performance GPUs, the motherboard must be capable of splitting PCIe lanes. In this setup, an Asus Prime X570-Pro was used to split the 16x PCIe lanes into 2x8. The RTX 5080 is connected via a high-quality PCIe 4 riser to the second slot.

VRAM Capacity

The combined VRAM of the RTX 3090 (24GB) and RTX 5080 (16GB) provides a total of 40GB. This allows the Qwen 3.6 27B Q8 quantization to fit entirely in VRAM while maintaining a 230k context window and Q8 KV-cache quantization.

BIOS and System Setup

Correct BIOS configuration is critical for multi-GPU stability and performance. The system must boot in UEFI mode; booting in BIOS/MBR mode prevents the use of both cards without complex kernel parameter modifications.

Required BIOS Settings

  • CSM (Compatibility Support Module): Disabled
  • Above 4G Decoding: Enabled
  • PCI Subsystem Settings: Enabled
  • ReSize BAR Support: Auto or Enabled
  • PCIEX16_1 Link Mode: Gen 4
  • PCIEX16_2 Link Mode: Gen 4

Driver and Kernel Configuration

For setups using GPUs from different generations (e.g., Ampere and Blackwell), the nvidia-open driver is the recommended choice. While patched drivers like open-gpu-kernel-modules exist, they often fail when mixing different GPU architectures.

To verify the connection and P2P status, the nvidia-smi topo -p2p r command can be used. For users with identical cards, the nova driver may be used, provided nvidia-dkms-open is uninstalled and the nova driver is blacklisted to ensure the patched driver loads at boot.

llama.cpp Implementation

Achieving high throughput requires specific build flags and runtime parameters to support multiple GPU architectures simultaneously.

Build Flags

To enable support for both the Ampere (RTX 3090) and Blackwell (RTX 5080) architectures, llama.cpp must be compiled with the following CMAKE_CUDA_ARCHITECTURES flag:

cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DGGML_NATIVE=ON -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES="86;120" -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc -DGGML_CUDA_NCCL=OFF

Runtime Optimization Parameters

The following llama-server configuration is used to maximize performance:

  • Speculative Decoding: --spec-type ngram-mod,draft-mtp --spec-draft-n-max 3 utilizes MTP speculative boost with ngram hints.
  • Multi-GPU Strategy: -sm tensor enables tensor splitting across GPUs.
  • VRAM Distribution: -ts 2,3 sets the card usage ratio to ensure VRAM is fully utilized across both GPUs.
  • KV Cache: -ctk q8_0 -ctv q8_0 --kv-unified optimizes memory usage for the context window.

Performance Results

Using the Huihui-Qwen3.6-27B-abliterated-ggml-model-Q8_0.gguf model, the setup achieves an average of 81.84 to 91.13 tok/s during decoding. Prompt evaluation speeds reach approximately 77.36 tok/s.

Power and Thermal Considerations

Community feedback highlights that this configuration is power-intensive, drawing approximately 700W at full load. Users should ensure a high-quality Power Supply Unit (PSU) and monitor power plugs for heat buildup, as the total system draw can approach 1kW.

Community Insights and Alternatives

Discussion among users suggests several alternative optimizations and model choices:

  • Model Alternatives: Some users recommend the heretic abliterated Qwen 3.6 27B model over the huihui version for better quality.
  • Tuning Parameters: For thinking mode, recommended settings are --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00. For coding, --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 is suggested.
  • Speculative Tuning: Some users suggest that --spec-draft-n-max 2 is more stable on NVIDIA hardware than 3.
  • Hardware Alternatives: Other users reported similar 80 tok/s performance using two RTX 3080 (20GB) cards on a MACHINIST X99 motherboard with Xeon CPUs for increased PCIe lane support.

Sources