Anthropic postmortem of August‑September 2024 Claude infrastructure bugs

TL;DR

Anthropic disclosed three separate infrastructure bugs that intermittently degraded Claude’s response quality between early August and early September 2024, and described the root causes, remediation steps, and upcoming improvements to prevent recurrence.

How Claude is Served at Scale

Claude is deployed across multiple cloud providers (AWS, Amazon Bedrock, Google Cloud Vertex AI) and hardware back‑ends (AWS Trainium, NVIDIA GPUs, Google TPUs). Anthropic enforces strict equivalence standards so that a request receives the same quality regardless of the underlying platform. This multi‑platform strategy introduces complexity: any code or configuration change must be validated on every hardware stack.

Timeline of the Incidents

Date	Event
Aug 5	First bug (context‑window routing error) introduced, affecting ~0.8 % of Sonnet 4 requests.
Aug 25‑26	Two additional bugs deployed (output corruption on TPU servers; approximate top‑k XLA miscompilation).
Aug 29	Load‑balancing change unintentionally amplified the routing error, pushing impact up to 16 % of Sonnet 4 requests in a single hour.
Sep 2	Output‑corruption bug rolled back.
Sep 4	Routing‑error fix deployed; approximate top‑k bug rolled back for Haiku 3.5.
Sep 12	Approximate top‑k bug rolled back for Opus 3.
Sep 16‑18	Full rollout of routing fix across first‑party platform, Vertex AI, and Bedrock.

The overlapping nature of these bugs made diagnosis difficult, as different symptoms appeared on different platforms and at varying frequencies.

The Three Overlapping Issues

1. Context‑Window Routing Error

What happened: Requests for Sonnet 4 with short context were mistakenly sent to servers configured for the upcoming 1 M‑token context window.
Impact: Initially 0.8 % of requests; peaked at 16 % of Sonnet 4 traffic on Aug 31. About 30 % of Claude Code users experienced at least one misrouted request. Misrouting on Amazon Bedrock peaked at 0.18 % of Sonnet 4 requests; on Vertex AI it stayed below 0.0004 %.
Why it mattered: Routing is “sticky”; once a session hit the wrong server, subsequent turns stayed on that server, compounding degradation.
Resolution: Fixed routing logic on Sep 4; completed rollout to all platforms by Sep 18.

2. Output Corruption on TPU Servers

What happened: A misconfiguration introduced a runtime optimization that assigned high probability to tokens that should be extremely unlikely, leading to nonsensical characters (e.g., Thai or Chinese glyphs) or syntax errors in code.
Impact: Affected Opus 4.1 and Opus 4 from Aug 25‑28, and Sonnet 4 from Aug 25‑Sep 2. Third‑party platforms (Bedrock, Vertex AI) were unaffected.
Resolution: Rolled back the change on Sep 2 and added automated detection tests for unexpected character outputs.

3. Approximate Top‑k XLA:TPU Miscompilation

What happened: A code change intended to improve token‑selection triggered a latent bug in the XLA:TPU compiler’s approximate top‑k operation, causing it to return incorrect high‑probability tokens for certain batch sizes and model configurations.
Impact: Confirmed on Claude Haiku 3.5; suspected on subsets of Sonnet 4 and Opus 3. No effect on third‑party platforms.
Resolution: Rolled back the change for Haiku 3.5 on Sep 4 and for Opus 3 on Sep 12. Switched to exact top‑k with fp32 precision, eliminating the bug at a modest performance cost. Collaborated with the XLA team on a permanent compiler fix.

Deep Dive: The XLA Compiler Bug

Claude’s generation pipeline uses top‑p sampling. On TPUs, probability calculations are performed across multiple chips, requiring coordinated sorting of token probabilities. The system relies on mixed‑precision arithmetic (bf16 for model math, fp32 for vector processing). The xla_allow_excess_precision flag, enabled by default, allowed the compiler to promote certain bf16 operations to fp32 for speed.

When the approximate top‑k optimization was applied, the precision mismatch caused the highest‑probability token to be dropped entirely for specific batch configurations. This manifested as random, hard‑to‑reproduce token‑selection errors, sometimes inserting foreign‑language characters into English responses.

Anthropic’s fix removed the earlier December 2024 workaround (which masked the symptom) and replaced the approximate top‑k with an exact implementation, restoring deterministic token selection at a slight efficiency penalty.

Why Detection Was Difficult

Evaluation gaps: Standard benchmark and safety evaluations did not surface the degradation because Claude can often recover from isolated errors.
Privacy constraints: Internal privacy controls limit engineers’ access to raw user interactions, hindering root‑cause analysis of undocumented failures.
Signal noise: Multiple bugs produced divergent symptoms across platforms, appearing as random quality fluctuations rather than a single systematic issue.
Load‑balancing side‑effects: The Aug 29 load‑balancing change amplified the routing error, creating a sudden spike that was not immediately linked to the underlying code change.

What Anthropic Is Changing

More sensitive evaluations – New tests that can reliably distinguish between correct and broken implementations will be continuously refined.
Production‑level quality monitoring – Evaluations will run on live production systems, not just internal canaries, to catch platform‑specific regressions.
Faster debugging tooling – Infrastructure will be built to safely inspect community‑sourced feedback without compromising user privacy, reducing remediation time for future incidents.
User‑feedback loops – Anthropic encourages users to report anomalies via the /bug command in Claude Code or the thumbs‑down button in Claude apps, and to share custom evaluation scripts with the team.

Implications for Users and the Industry

Reliability expectations: Even with rigorous infrastructure, complex multi‑platform deployments can introduce subtle bugs that affect model output quality.
Transparency value: Anthropic’s detailed postmortem provides a model for how frontier AI labs can disclose infrastructure failures, fostering trust and enabling the community to contribute to detection.
Tooling importance: The incident underscores the need for precise, privacy‑preserving debugging tools and continuous production monitoring in large‑scale LLM services.
Future safeguards: By moving to exact top‑k sampling and expanding evaluation coverage, Anthropic aims to raise the baseline reliability of Claude across all hardware back‑ends.

Anthropic thanks the community for its feedback and invites further bug reports and evaluation contributions via feedback@anthropic.com.