Qwen3.8 27B Quantization Benchmarks: 4-bit Matches BF16, 1-bit Fails

TL;DR

Qwen3.8 27B quantized to 4‑bit (Q4_K_M, 17 GB) delivers essentially the same benchmark scores as the full BF16 model, fitting comfortably on a 24 GB GPU. Quantizations below 4‑bit degrade performance, with 1‑bit (UD‑IQ1_S, 6.2 GB) dropping to random‑chance levels on GPQA Diamond and failing on coding tasks.

4‑bit Quantization Holds Up

Conclusion: The 4‑bit Q4_K_M quantization (17 GB) matches BF16 on GPQA Diamond, IFBench, and Terminal‑Bench 2.1, making it the sweet spot for consumer GPUs.

The full BF16 model requires ~55 GB VRAM, exceeding most desktop cards.
Q4_K_M runs on an RTX 4090 (24 GB) while leaving ~64 k tokens of KV‑cache.
On GPQA Diamond, scores for BF16, 8‑bit, 4‑bit, and even 2‑bit are statistically indistinguishable (Wilson 95 % confidence intervals overlap). Only the 2‑bit UD‑Q2_K_XL shows a modest dip.
IFBench shows no measurable difference between BF16 and 2‑bit models, though the context window is limited to ~4 k tokens.
Terminal‑Bench 2.1 (89 agentic coding tasks, 3 h timeout, xhigh effort, 98 k context) records identical success rates for BF16 and Q4_K_M. The 2‑bit model falls slightly but remains comparable to mid‑tier commercial agents (e.g., Opus 4.7, Gemini 3.1 Pro).

“The 17 GB Q4_K_M matches the full model on a popular agentic coding benchmark, Terminal‑Bench 2.1. It fits on a 24 GB card such as RTX 4090, still leaving room for about 64k tokens of context.” – Quesma blog

Why 4‑bit Works

Quantization method Q4_K_M (Unsloth v2) preserves weight distribution better than naive 4‑bit schemes.
The author used a fixed F16 KV‑cache (≈2.3 GB per 32 k tokens), so memory savings come solely from weight compression.
Higher reasoning effort (xhigh) compensates for any minor probability shifts introduced by quantization, as noted by commenters.

2‑bit Quantization Is Usable but Weaker

Conclusion: The 2‑bit UD‑Q2_K_XL (10.7 GB) remains functional on most benchmarks but incurs a small performance penalty.

GPQA Diamond scores dip modestly compared to 4‑bit.
IFBench still shows no significant drop, suggesting that many instruction‑following tasks are tolerant to 2‑bit noise.
On Terminal‑Bench 2.1, success rates fall a few points but stay within the range of older commercial agents.
Token usage increases: for solved tasks, the 2‑bit model writes ~25 % more tokens than BF16 while keeping the number of reasoning turns roughly constant.

1‑bit Quantization Collapses

Conclusion: 1‑bit UD‑IQ1_S (6.2 GB) fails dramatically, scoring near random guessing on GPQA Diamond and performing poorly on coding benchmarks.

Scores drop below the random‑guess baseline, especially at higher reasoning effort (xhigh), where longer generation leads to empty answers.
The author’s measurements contradict Unsloth’s claim of “72 % top‑1 accuracy” – the missing 28 % is critical for task success.
Community reports (e.g., r/LocalLLaMA) echo the same failure mode, describing the model as “brain‑damage quant.”

“While 2‑bit quantizations work to some extent, even the best 1‑bit model is useless for these benchmarks.” – Quesma blog

Technical Insight

1‑bit quantization on a pre‑trained dense model is fundamentally unstable; without quantization‑aware training (QAT) on massive data, the weight representation collapses.
As one commenter notes, “Quantization on an existing pre‑trained model will almost always collapse at 1‑bit.” (HN comment)

Benchmark Methodology

Conclusion: The author reproduced official BF16 scores before testing quantizations, ensuring a reliable baseline.

Benchmarks used: GPQA Diamond (graduate‑level science QA), IFBench (instruction following), Terminal‑Bench 2.1 (agentic coding).
Reasoning effort levels: low, medium, xhigh (default). Higher effort generally improves scores but also increases token consumption.
Hardware: NVIDIA L40S (48 GB), H100 (80 GB), H200 (141 GB). Modal GPU rentals were used, costing roughly $3 k for the full suite.
Model loading employed llama.cpp (build from 16 Aug 2026) with F16 KV‑cache regardless of weight quantization.
Confidence intervals shown are Wilson 95 % intervals; a comment on HN clarifies they are conservative and not directly tied to run‑to‑run variance.

Community Perspectives

Conclusion: Commenters largely confirm the findings, add nuance about reasoning effort, and raise questions about KV‑cache quantization and GPU memory limits.

Reasoning effort matters: One commenter observes that longer thinking can offset quantization noise, albeit at the cost of more tokens.
GPU memory ceiling: Users note that 4‑bit models still exceed the capacity of 16 GB cards for long contexts; 3‑bit or mixed‑precision schemes could fill that gap.
KV‑cache quantization: A request for benchmarks combining model quantization, KV‑cache quantization, and context size highlights an unexplored dimension.
Hardware practicality: Several comments point out that many consumers have 8‑12 GB GPUs, where even 4‑bit may be too large for useful context lengths.

Cost Considerations

Conclusion: Running full‑scale benchmarks on cloud GPUs is expensive; 4‑bit quantization reduces both memory and compute cost, but the savings are modest compared to API pricing of large MoE models.

Modal GPU rentals were used; total spend ≈ $3 000.
For reference, DeepSeek V4 Flash 0731 (284 B MoE) costs ~ $0.10 per million output tokens on OpenRouter, far cheaper per token than running a dense 27 B model locally.
The memory savings of 4‑bit (≈ 38 GB vs. 55 GB) translate to the ability to run on a single RTX 4090, avoiding multi‑GPU setups.

Practical Recommendations

Conclusion: For most local workloads, use the 4‑bit Q4_K_M quantization; 2‑bit is acceptable for low‑resource contexts; avoid 1‑bit entirely.

Select the highest quantization that fits your GPU while preserving the required KV‑cache size for your context window.
Set reasoning effort to xhigh if you notice minor accuracy drops; expect higher token usage.
Monitor KV‑cache memory – even with a 4‑bit model, a 32 k token cache consumes ~2.3 GB; longer contexts may require larger GPUs.
Consider mixed‑precision or 3‑bit schemes if you need >64 k tokens on a 16 GB card (not covered in this benchmark).
Avoid 1‑bit quantization unless you perform quantization‑aware training on massive datasets, which is currently impractical for most users.

Final Thoughts

Quantization of Qwen3.8 27B is practically safe down to 4‑bit, delivering BF16‑level quality on a wide range of tasks while fitting on consumer‑grade GPUs. The performance curve is non‑linear: little change from BF16 to 4‑bit, a modest dip at 2‑bit, and a catastrophic collapse at 1‑bit. Embracing quantization—especially the well‑engineered Q4_K_M variant—enables local deployment of large language models without sacrificing capability.