Cloudflare Workers AI: Running Kimi K2.6 and GLM 5.2 at Scale with KV Cache Quantization, INT4 Weight Compression, and Integrity Checks
Quantizing the KV cache cuts memory use and raises concurrent request capacity
Storing the KV cache in FP8 (e4m3) instead of BF16 halves its size, allowing roughly double the context length on Kimi K2.6—from about 686,000 tokens to 1.37 million tokens—while keeping model answers unchanged.
| Concurrent requests | BF16 KV cache (tok/s) | FP8 KV cache (tok/s) |
|---|---|---|
| 1 | 137 | 125 |
| 8 | 731 | 689 |
| 16 | 1,106 | 1,028 |
| 32 | 1,558 | 1,489 |
| 64 | Out of memory | 2,192 |
At any single concurrency level BF16 is a few percent faster per token, but BF16 runs out of cache at 32 concurrent requests whereas FP8 sustains up to 64 concurrent requests, reaching 2,192 tok/s—about 41% higher than BF16’s peak and roughly 30% lower cost per token. Accuracy benchmarks show no meaningful difference:
| Benchmark | BF16 KV | FP8 KV |
|---|---|---|
| GSM8K | 94.24 | 94.09 |
| ARC-Easy | 89.06 | 89.14 |
| ARC-Challenge | 66.72 | 67.49 |
| MMLU | 89.11 | 89.04 |
| MMLU-Pro | 80.29 | 79.29 |
| mcxams (internal) | 61 / 63 | 61 / 63 |
| Tool-call validity | 92.2% | 92.6% |
Because prefill is compute-bound, Cloudflare keeps the KV cache in BF16 for prefill and uses FP8 only for decode, where the memory bound dominates.
Compressing model weights to INT4 reduces GPU memory and speeds decode
For GLM 5.2, compressing weights from FP8 to INT4 shrinks the checkpoint from 705 GB to 421 GB (~40% reduction) and cuts per‑GPU memory in an 8‑way tensor‑parallel deployment from ~88 GB to ~52 GB, freeing space for about 1.18 million tokens of KV cache on the same hardware.
Accuracy remains within 0.8 points of the FP8 model across all benchmarks:
| Benchmark / Capability | Metric | FP8 | INT4 |
|---|---|---|---|
| GSM8K | Exact match | 94.39% | 93.56% |
| GSM8K | Flexible | 94.24% | 93.48% |
| ARC-Easy | Accuracy | 86.62% | 86.15% |
| ARC-Easy | Acc (norm) | 84.51% | 85.19% |
| ARC-Challenge | Accuracy | 64.93% | 64.85% |
| ARC-Challenge | Acc (norm) | 67.24% | 66.64% |
| MMLU | Average | 86.60% | 86.54% |
| MMLU-Pro | Exact | 80.80% | 80.47% |
| mcxams (internal) | Passed | 62 / 63 | 62 / 63 |
Decode throughput improves because less weight data must be streamed from memory:
| Concurrent requests | GLM FP8 (tok/s) | GLM INT4 (tok/s) | INT4 gain |
|---|---|---|---|
| 1 | 60 | 92 | +55% |
| 8 | 425 | 513 | +21% |
| 16 | 683 | 825 | +21% |
| 32 | 994 | 1,267 | +27% |
| 64 | 1,672 | 1,933 | +16% |
Prefill is compute‑bound; expanding INT4 weights back to FP8 makes prefill slower (10,160 tok/s FP8 vs 8,660 tok/s INT4). Cloudflare therefore runs INT4 for decode and FP8 for prefill, exploiting the disaggregated prefill/decode pools.
Protecting a shared KV cache adds integrity checks with negligible overhead
More requests sharing GPU memory raises the risk of cache‑page corruption. Cloudflare added a KV cache integrity layer: each physical page gets a tag that changes on reallocation, and the server verifies request‑to‑page mappings before decode reads. Mismatches abort the request instead of returning wrong data.
Overhead measurements on a mid‑sized production model (2 prefill, 2 decode, 8,192‑token inputs, 1,000‑token outputs):
| Concurrency | Throughput change | p95 latency change |
|---|---|---|
| 1 | −0.53% | +0.42% |
| 2 | −0.38% | +0.54% |
| 4 | −0.79% | +0.63% |
| 8 | −0.43% | +0.80% |
The cost stays under 1% for both throughput and tail latency. The check runs as a separate batch to avoid GPU thread‑group races, is enabled per deployment, and defaults to a no‑op tracker with zero measurable overhead for deployments that do not need it.
What’s next
Cloudflare is expanding FP8 KV caches across more of its fleet, evaluating NVFP4 weight formats on Blackwell GPUs, and working to make integrity checks universally enabled at negligible cost. These efforts aim to support more customers at lower cost while preserving model accuracy.
Community feedback
Nice to see a provider being transparent about KV cache quantisation. I've been suspecting that some providers do this silently whilst heavily promoting their unquantised weights, even though KV quantisation can degrade quality more than weight quantisation. However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Secondly, the evaluation suite they use to claim that FP8 KV quantisation is indistinguishable is noticeably lacking coding benchmarks; in long-running tasks, minor tool call errors compound over time. — @scrlk
Why int4? There are a lot of superior 4 bit formats like nf4 from bitsandbytes. — @om8
I was interested in reading this until my slop detector went off at the paragraph starting with “It's worth being precise about where the benefit comes from, because it isn't raw speed.” I love AI, but I really hate reading it. — @brokenodo
So they quantize models, only tell about it in the blog post (instead of a warning on the model page), and even in the blog post pretend there's no difference by benchmarking on small context tasks many of which are saturated. Coding agents will probably be severely negatively affected by KV quantization. I'd say serving quantized models without saying so on the