Cross-Request KV Cache Exposure in Cloudflare Workers AI: GLM-5.3, SGLang, and the Fix in v0.5.22
Between August 27 and September 9, 2026, a small number of customers using the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) received responses that contained data from other customers' requests to the same model. The cause was two overlapping problems: a Cloudflare configuration mismatch between the model's advertised 1,048,576-token context window and a prefill KV cache pool that only held 681,280 tokens, and an SGLang bug that let over-length requests pass through the disaggregated prefill-to-decode handoff instead of being rejected. In total, 155 responses were affected, all to prompts longer than 681,280 tokens. Cloudflare remediated the issue and restored the model on September 9, 2026, and confirmed that no other model hosted on Workers AI was affected.
What happened
Cloudflare was alerted on September 8, 2026 by security researcher Gadi Evron of Knostic, who reported through Cloudflare's HackerOne program that very long prompts sent to @cf/zai-org/glm-5.3 on Workers AI sometimes returned text belonging to other users' sessions. Cloudflare reproduced the issue the next morning, declared a security incident, and took the model offline within 40 minutes of that declaration. The company then deployed a corrected configuration and a patched inference engine, and restored service the same day.
Cloudflare states that after excluding its own testing and the researcher's, the affected requests are consistent in timing and volume with ordinary long-context use rather than repeated exploitation, and that it found no evidence of deliberate exploitation.
GLM is a text generation model from Z.ai built for agentic coding workflows, and it is one of the models available through Workers AI. The exposure was present from the model's launch on Workers AI on August 27, 2026.
Issue one: the KV pool was smaller than the context window
Cloudflare deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on its prefill GPUs only had room for 681,280 tokens. Cloudflare was aware of this mismatch before launch.
SGLang sizes the KV cache pool at startup from whatever GPU memory remains after loading model weights. The pool size is not a configuration value, and the model's advertised context length is a separate configuration value. Nothing forces the two to agree, and Cloudflare's post states that checking that agreement is the operator's job. The selected KV pool size is printed by SGLang in startup logs and exposed as a metric.
Cloudflare's pre-launch testing observed successful HTTP status codes and a streaming body with no error frames during benchmarking, so the company assumed the engine was handling the difference between the KV cache pool and the requested context window gracefully. The post identifies this as a critical testing gap: Cloudflare validated HTTP response codes and engine performance at long contexts but did not validate model output and coherence.
Issue two: the SGLang rejection path in disaggregated mode
In disaggregated prefill/decode mode, a prompt longer than the KV pool could hold was internally marked as rejected and replaced with a one-token placeholder, but the request was not aborted. Instead, prefill generated the KV cache for the placeholder prompt and transferred it to the decode node. The decode node received a "transfer complete" message and assumed prefill had overwritten the KV cache from previous requests with the full prompt for this request, when in fact most of the previous state remained intact. Decode then generated a response while attending over stale cache pages that still held data from earlier requests.
On a single-node deployment the rejection path worked correctly: the placeholder ran, the batch finished, and the client received an error. In disaggregated mode, the admission queues on both prefill and decode checked whether a request was finished, not whether it was rejected. A rejected request stored its rejection in to_finish rather than finished_reason, so finished() still returned False and the request reported itself as live.
The sequence Cloudflare describes is:
- Bootstrap. The router told the decode node the prompt length (8 pages in Cloudflare's diagram), and decode reserved 8 KV pages.
- Prefill admission. The prefill admission queue admitted any request where
req.finished()was False, so the rejected request was admitted. - Prefill forward pass. Rejection had already replaced the prompt with a one-token placeholder, so prefill ran a trivial forward pass on that one token.
- KV transfer. Prefill sent one page of KV to decode and signaled that the transfer was complete.
- Decode admission. Decode's admission queue used the same
finished()check and accepted the handoff. - Decode attention. Decode started generating and attention read all 8 reserved pages. Only page P0 had been written by this request; pages P1 to P7 still held what earlier requests left in them.
- Response. The client received HTTP 200 instead of HTTP 400.
prompt_tokensmatched the long prompt and the reply was coherent, but it arrived faster than an 8-page prefill should take and contained content from other users' KV pages.
Every request over the threshold behaved this way. The bug had been reported to SGLang by another user as issue #30233 on July 6, 2026, which Cloudflare noticed while investigating this incident. Cloudflare states it had been flagging these types of issues with SGLang since v0.5.10.
GLM 5.3 was the only model affected. Every other disaggregated deployment on Workers AI has KV pools larger than its context length on both prefill and decode, so no request could reach the rejection path.
Remediation: fix the configuration and fix the engine
Cloudflare applied two independent fixes before restoring public access.
Make the pool hold the full context. Cloudflare adjusted its GLM 5.3 configuration to match its GLM 5.2 configuration, freeing enough GPU memory that the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context length. A prompt that fits the context window now always fits the pool, so the rejection path is never reached for a valid request.
Make the rejection path safe. Cloudflare argues that relying on pool size alone is fragile, because changes to quantization, tensor parallelism, CUDA graph sizes, or a future model version change the pool size in a way that is easy to miss, and because other situations may let the prefill scheduler reject a request and trigger the same bug. Cloudflare patched the engine so a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer happens. It proposed this upstream as #38704, which also had the decode node verify that the number of tokens transferred matches the number it reserved.
Cloudflare validated both changes against the researcher's reproduction script and its own canary-marker tests before restoring the model.
Timeline of the incident
All times are UTC, as published by Cloudflare.
| When | What |
|---|---|
| Aug 27, 2026 | GLM 5.3 launched on Workers AI with a prefill KV pool smaller than its context window. The exposure was present from launch. |
| Sep 8, 2026 – 23:11 | Report received via HackerOne. |
| Sep 9, 2026 – 08:00 | Cloudflare begins verifying the HackerOne report. |
| Sep 9, 2026 – 13:22 | Security incident declared. Product Security reproduced the leak using two independent accounts within 25 minutes. |
| Sep 9, 2026 – 14:00 | GLM 5.3 set to private. Requests return HTTP 400. Status page updated. |
| Sep 9, 2026 – 14:35 | Root cause identified as the prefill pool being smaller than the context length. Canary prefill node with a larger pool confirmed not susceptible. |
| Sep 9, 2026 – 14:52 | All prefill nodes rolled to the corrected configuration. |
| Sep 9, 2026 – 19:04 | Full deployment rebuilt with the corrected configuration plus a Cloudflare-authored engine patch. Validated against the researcher's reproduction script. |
| Sep 9, 2026 – 19:40 | Model restored to public. |
| Sep 10, 2026 – 23:30 | Meeting with the SGLang maintainers (RadixArk CTO and core engineers) to agree on upstream fixes. |
| Sep 11–19, 2026 | Additional hardening rolled to all production SGLang deployments. |
| Sep 16–Oct 12, 2026 | Upstream SGLang merges the admission fix, per-transfer checksums, abort-ACK preservation, and turns deferred KV release on by default. |
| Oct 2, 2026 | SGLang v0.5.21 released. |
| Oct 19, 2026 | SGLang v0.5.22 released. |
| Oct 26, 2026 | Coordinated publication with the researcher and RadixArk. |
Upstream SGLang fixes
The patch for this issue was released in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22. Cloudflare recommends that all operators update SGLang dependencies to v0.5.22 or later. The following changes landed on SGLang main; all but one are included in v0.5.22.
| Category | Pull request | Release | What it does |
|---|---|---|---|
| Incident | #36988 | v0.5.20 | Retires aborted prefill requests before their result reaches decode. This stops the exposure in this incident. |
| Incident | #38935 | v0.5.21 | Rejects these requests at admission, before any bootstrap, allocation, or KV transfer. Follows the approach in Cloudflare's draft patch, #38704. |
| Race | #39461, #41612 | v0.5.21, v0.5.22 | sgl-router cancels both halves of a request on client disconnect or prefill failure. |
| Race | #37454, #41023 | v0.5.20, v0.5.21 | Enables deferred KV release only on backends that support it, then turns it on by default. |
| Race | #40645, #41402 | v0.5.21 | Fixes cases where the drain confirmation was lost, so releases happen on confirmation instead of waiting out the timeout. |
| Race | #41404 | v0.5.21 | Holds pages on every transfer failure, not only decode-initiated aborts. |
| Race | #41618 | v0.5.21 | Fixes a regression in the abort notification path. |
| Race | #43719 | v0.5.22 | A failed Mooncake send no longer reports its writes as drained. |
| Race | #37077 | v0.5.22 | Tags abort messages per request so a reused ID cannot consume a stale confirmation. Adds MORI support. |
| Race | #36612 | v0.5.20 | Shares the prefill-to-decode failure notification across Mooncake, NIXL, and MORI. |
| Race | #32758 | v0.5.20 | Optional cap on Mooncake transfer batch size, so one timed-out call cannot leave an unbounded number of writes in flight. |
| Lifetime | #30899 | v0.5.21 | Decode waits for an in-progress offload to host memory before retracting a request and freeing its pages. Authored by Cloudflare. |
| Lifetime | #43562 | v0.5.22 | Keeps tensor-parallel ranks in agreement when a host-memory cache restore fails. |
| Observability | #39500 | v0.5.21 | Adds --disaggregation-enable-kv-checksum: decode verifies a checksum of the transferred KV before generating. Cleanup in #39729, merged and awaiting release. |
| Observability | #38596, #40642 | v0.5.20, v0.5.21 | Fixes for SGLang's KV canary, which detects and logs potential corruption. |
| Observability | #41128 | v0.5.22 | Metrics for deferred release: count, duration, and whether each hold ended on drain confirmation or timeout. |
Still in progress at the time of publication: #39729 (cleanup for --disaggregation-enable-kv-checksum, merged and awaiting release), zeroing KV pages on release (draft proposal #46435), extending KV checksum validation from the prefill-to-decode handoff to every decode step, and broader KV Canary and KV Checksum model and feature support.
Additional hardening Cloudflare deployed
Cloudflare classifies this bug as one of a family of KV cache memory-safety bugs it has been working on for several months. In each case, a page is freed, reused, or read while something else still has a claim on it. Cloudflare identifies two common causes.
The first is page lifetime within a single node. Several tasks on a decode node can hold the same page at once: token generation, offload to host memory, the KV transfer receiver, and the abort and error paths. If one frees the page while another is still using it, a new request can be given that page while it is still being read or written. Earlier fixes for decode offload (#24226, #30899) and double release on grammar errors (#30937) were of this kind.
The second is a race across nodes. When a request is aborted while prefill is still writing its KV cache to decode over RDMA, the writes already sent cannot be recalled. If decode frees those pages right away, those pages can be given to another request, and the late writes land in that request's KV cache. Cloudflare contributed proposed fixes (#31357, #38961) and a reproduction script (#32564) for this race. Cloudflare notes that the bug in this incident was different: a rejected request went through the handoff every time rather than by a timing accident, but it had the same result of decode reading pages the request never wrote.
Since the incident, Cloudflare has:
- Extended deferred release to hold prefill's source pages as well, and to cover every failure type, not just aborts. Proposed upstream as #38961, superseded by #40645, #41402, and #41404.
- Deployed bounded Mooncake transfer batches (#32758, from the SGLang community), so no single transfer call can outlast the release timeout.
- Changed its load balancer so a client disconnect no longer aborts a request mid-transfer. It holds the prefill connection until prefill finishes and the decode connection until the first token, then closes both.
- Added a check to every disaggregated deployment change: KV pool size measured against context length, deferred release on both roles, transfer batch bounds inside the release timeout, and all required patches applied.
What Cloudflare recommends to other SGLang operators
Cloudflare's guidance for anyone running SGLang in disaggregated prefill/decode mode is:
- Measure your KV pool and compare it to your context length, on every prefill and decode node, from the engine's startup log, after every change to weights, quantization, parallelism, or memory fraction. If the pool is smaller, lower the context length to fit. Do not rely on a proxy to enforce the limit, and do not assume the engine will reject what does not fit.
- Upgrade to a build that includes #36988 and #38935 (v0.5.22 or later), so an over-length request is rejected before handoff regardless of pool size.
- Run with deferred KV release on both roles (default after #41023), and bound Mooncake transfer batches so a single timed-out call cannot leave an unbounded tail of writes.
- Turn on corruption detection where the model supports it.
--disaggregation-enable-kv-checksumand--kv-canarydo not yet cover every model architecture. Check logs to confirm requests are being verified. Cloudflare is working with the SGLang team to extend both checks to more architectures, including GLM 5.3, GLM 5.3 Flash, and DeepSeek V4. - Ensure your proxy is not generating unnecessary aborts. Configure the load balancer to allow requests to proceed up to the first token, so it does not trigger an abort that requires cross-node coordination.
Longer term: integrity by construction and formal verification
Cloudflare argues that configuration checks and lifecycle fixes reduce the number of ways a page can be misused but do not make misuse impossible. The durable answer, in its framing, is for the engine to detect at read time that a request is about to attend over a page it does not own, and fail that one request rather than return someone else's data.
Cloudflare built a prototype of this over the summer and described it in August 2026. Every physical KV page carries a tag that changes when the page is reallocated, each request records the tags it expects, and the mapping is checked before decode reads the cache. In Cloudflare's tests it cost under 1% in throughput and tail latency. Cloudflare tested it on GLM 5.2 in mid-August but disabled it shortly afterwards due to reliability issues. The prototype was also tied to specific attention backends and model architectures, so it could not be a general fix. Cloudflare says it is working with the SGLang team to build a production version into the engine that works across every model. The per-transfer checksum in #39500 is a first step because it validates the handoff from prefill to decode; page-level tags go further because they check every read, including reads long after the transfer.
Cloudflare also frames the bug as a consistency failure in a distributed system: RDMA copies data directly from one node's memory to another's, and the two machines must agree on where the data goes, how large it is, and when it is finished. Cloudflare points to its existing formal-verification work — Topaz for conflict-free IP selection in authoritative DNS, and models of Meerkat's QuePaxa implementation that demonstrated consistency bugs later fixed — and says it is now directing that experience at KV cache transfer engines. The stated plan is to model and prove the safety of an RDMA-based KV cache transfer protocol, model its production SGLang distributed KV cache system using Mooncake, and track and fix any divergence from the safety properties it wants to guarantee.
What the Hacker News discussion added
The most repeated criticism in the Hacker News thread was that the title over-attributes the bug to the model. Commenter jacquesm wrote:
This doesn't really have anything to do with GLM-5.3. This is what happens when you share models between users and don't do your bookkeeping properly, the actual model doesn't matter. What it shows is that there isn't enough isolation between the user sessions and that memory can overlap when that should never be possible. This is an SGLANG and a Cloudflare bug long before it is a Z.AI model bug, I don't see what they could have done about it. Cloudflare should own it instead of naming other (imo innocent) parties in the title.
armcat proposed the alternative framing "Cross-Tenant KV Cache Exposure in Shared LLM Inference: Analysis and Remediation," adding that as a GLM-5.3 user they initially thought there was cause for concern "only to find out there isn't." sauravadhk and adg001 made similar suggestions to drop the model name from the title.
Several commenters focused on the testing gap Cloudflare itself disclosed. ComputerGuru wrote:
Surely "we expected and error but got a successful response" merits investigating rather than moving on?
segmondy raised the related question of whether Cloudflare advertised a 1,048,576-token context window while only having room for 681,280 tokens.
Others praised the disclosure itself. kami23 wrote that they liked the root-cause analysis "a lot more than most," citing the ownership and the clear explanation of why testing did not catch the bug, and the speed at which Cloudflare could disable serving of the model.
The thread also generalized the problem beyond this incident. 0xbadcafebee noted that shared tenancy means tokens cannot be magically isolated, and that "enterprises with highly sensitive data should really be on non-shared hosts." totetsu asked whether anyone offers single-tenant inference as a service. nitishr wrote they were "surprised more serverless providers haven't found (or publicly disclosed) similar issues, yet," and thehamkercat suggested other providers "probably don't even know/care that this is happening."
Two commenters described what they believe are similar cross-tenant leaks elsewhere. xmcqdpt2 recounted a Claude response at work that returned "very specific, unrelated information that looked very much like it could have come from someone else in the company," and said Anthropic told them it wasn't possible and that AI sometimes makes mistakes. reassess_blind described finding that a browser testing service's iOS VMs leaked clipboard data between users, including another user's credentials, and said the vendor declined a bounty. These are unverified anecdotes from commenters, not findings from Cloudflare's report.
One commenter, octoberfranklin, characterized the incident as a case where "even though sglang is written in Rust, they still got burned by undefined behavior from uninitialized memory." Cloudflare's own report describes the root cause differently: a rejected request was admitted because the admission queues checked finished() rather than rejection state, so decode attended over stale KV pages that had not been overwritten.
Coordinated disclosure
Cloudflare published the report as part of a coordinated disclosure alongside a post from the SGLang team. Cloudflare thanked Gadi Evron, who reported the issue through its HackerOne program with a reproducible script, and the SGLang team at RadixArk, which Cloudflare says treated the incident as its own problem and landed upstream fixes with urgency. Security issues in Workers AI can be reported at hackerone.com/cloudflare.