Qwen3.8‑27B MTP – a community‑discovered speed boost for llama.cpp

What it is

• A set of command‑line flags for llama.cpp that unlock speculative multi‑token‑prediction (MTP) decoding for the Qwen‑3.8‑27B model (distributed as a GGUF file). The flag combination (--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1) can make the model generate 33 %–145 % faster on consumer‑grade GPUs without any model conversion or custom builds.

Why it matters

• The speed gain comes from draft‑MTP: the server first drafts several tokens with a lightweight head and only runs the full model on tokens that fail verification. This reduces the number of expensive forward passes.
• The community has mapped out 53 different hardware configurations (from 2016 Pascal GPUs to 2026 Blackwell cards, Apple Silicon, and even unlocked mining cards) and distilled seven practical tuning rules that let users reproduce the boost on their own rigs.

How to use it

1. Download the GGUF for Qwen‑3.8‑27B (e.g. unsloth/Qwen3.8‑27B‑GGUF).
2. Run the reference command (adjust -c for context length if needed):

   llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
     -c 131072 -ngl 999 -fa 1 \
     --cache-type-k q4_0 --cache-type-v q4_0 \
     --spec-type draft-mtp --spec-draft-n-max 2 --parallel 1
   

3. Tune:
   • Sweep --spec-draft-n-max (2‑4) to find the sweet spot for your GPU.
   • On bandwidth‑limited machines, try --spec-draft-p-min 0.60‑0.75.
   • For multi‑GPU setups, set --split-mode tensor before applying the spec flags.
   • Always benchmark with --parallel 1 (the spec advantage disappears with many concurrent streams).
4. Measure the effect with the provided probe.py script, which streams tokens from the server and reports median per‑prompt speeds.

Key take‑aways from the community data

• GPU class matters: 24 GB cards (RTX 3090/4090) typically peak at n‑max = 2; newer, faster cards (RTX 5090, Blackwell) can push to n‑max = 4‑6.
• Split mode (layer vs tensor) can add another +68 % on a dual‑GPU pair.
• Parallelism: the spec speed‑up is a single‑stream optimization; using --parallel 4 removes the benefit.
• Memory pressure: on shared desktops the OS may spill weights to host RAM, halving speed. Run head‑less or verify that the full model stays resident (mem_info_gtt_used or vendor‑specific tools).
• Baseline vs. spec: across the 53 reported cards, the boost ranges from modest (+30 %) to dramatic (+180 %), especially on large context windows (256 K tokens) where the overhead of full passes dominates.

Who benefits

• Anyone running Qwen‑3.8‑27B locally with llama.cpp—researchers, hobbyists, or developers building AI‑augmented apps—who wants faster generation without buying new hardware.
• The repo also serves as a living benchmark collection; contributors can add new rows by opening a PR with their own measurements.

Caveats

• The speed gain is measured with single‑stream decoding; multi‑user servers will not see the same improvement.
• Prompt embedding transfers add a small overhead.
• Results depend on driver versions, thermals, and exact llama.cpp build; rebuilding llama.cpp before tuning is recommended.

---

TL;DR: By enabling the draft‑mtp speculative decoding flags in llama.cpp, the community has turned the Qwen‑3.8‑27B GGUF into a much faster model on a wide range of GPUs, and the repository documents the exact flag combos, tuning rules, and benchmark data needed to reproduce the gains.