Cloudflare Workers AI: Optimizing Kimi and GLM Inference at Scale

Cloudflare has implemented three primary optimization techniques—KV cache quantization, model weight compression, and cache integrity checking—to serve large, long-context mixture-of-experts models like Moonshot's Kimi K-series and Z.ai's GLM. By leveraging the SGLang inference framework and a disaggregated architecture that separates prefill and decode phases, Cloudflare increases request concurrency and throughput while maintaining model accuracy.

KV Cache Quantization for Increased Concurrency

Quantizing the Key-Value (KV) cache from 16-bit precision (BF16) to 8-bit floating point (FP8, e4m3) doubles the available context memory, allowing more concurrent requests to reside on a single GPU.

For the Kimi K2.6 model, FP8 quantization increases the memory capacity from approximately 686,000 tokens to 1.37 million tokens. While BF16 is slightly faster per token at low concurrency, FP8 enables significantly higher total system throughput by avoiding "out of memory" errors at higher loads. In benchmarks on a disaggregated H200 deployment, FP8 supported 64 concurrent requests reaching 2,192 tokens per second, whereas BF16 peaked at 32 concurrent requests.

Cloudflare applies this optimization selectively: the decode phase uses FP8 to maximize concurrency, while the prefill phase remains in BF16 because prefill is compute-bound rather than memory-bound.

Model Weight Compression for Lower Latency

Compressing model weights from 8-bit floating point (FP8) to 4-bit integers (INT4) reduces the memory footprint of the model, which directly accelerates the decode phase by reducing the amount of data streamed from GPU memory.

For GLM 5.2, weight compression reduced the checkpoint size from 705 GB to 421 GB. In an 8-way tensor-parallel deployment, per-GPU memory usage dropped from 88 GB to 52 GB, freeing space for an additional 1.18 million tokens of KV cache. This resulted in significant latency gains during the decode phase, specifically a 55% increase in tokens per second for a single concurrent request.

Because INT4 weights must be expanded before multiplication, the compute-bound prefill phase is slower in INT4 (8,660 tok/s) compared to FP8 (10,160 tok/s). Consequently, Cloudflare uses FP8 for prefill and INT4 for decode.

KV Cache Integrity Checking

Increasing the number of requests sharing a single GPU's memory increases the risk of bookkeeping errors in paged attention and continuous batching. To mitigate this, Cloudflare implemented a KV cache integrity checking layer.

This system assigns a tag to every physical cache page that changes upon reallocation. The server verifies these tags before decode operations read from the cache; if a mismatch is detected, the request is aborted to prevent the model from returning data from the wrong page.

Performance overhead for this safety check is negligible, with throughput and p95 latency changes remaining under 1% for mid-sized production models. The validation is run as a separate batch check to avoid race conditions within the attention kernel.

Performance and Accuracy Benchmarks

Cloudflare reports that neither KV cache quantization nor weight compression significantly impacts model quality across standard benchmarks.

Kimi K2.6 (BF16 vs FP8 KV Cache):

• GSM8K: 94.24 (BF16) vs 94.09 (FP8)
• MMLU: 89.11 (BF16) vs 89.04 (FP8)
• Tool-call validity: 92.2% (BF16) vs 92.6% (FP8)

GLM 5.2 (FP8 vs INT4 Weights):

• GSM8K (Exact match): 94.39% (FP8) vs 93.56% (INT4)
• MMLU Average: 86.60% (FP8) vs 86.54% (INT4)
• ARC-Challenge Accuracy: 64.93% (FP8) vs 64.85% (INT4)

Community Perspectives

While Cloudflare emphasizes the lack of accuracy loss, some community members on Hacker News raised concerns regarding the depth of the evaluation:

"some model families are more sensitive to KV quantisation than others... the evaluation suite they use to claim that FP8 KV quantisation is indistinguishable is noticeably lacking coding benchmarks; in long-running tasks, minor tool call errors compound over time."

Other critics argued that serving quantized models without explicit warnings on the model's landing page could be misleading to users who require maximum precision for complex tasks like coding agents.