IL-0022 Anthropic · 2025-08-05ai-infra
Between August and early September 2025 three separate infrastructure bugs intermittently degraded the quality of Claude's responses. Anthropic serves Claude through its own API, Amazon Bedrock and Google Cloud Vertex AI, on AWS Trainium, NVIDIA GPUs and Google TPUs, and the bugs affected these platforms differently. Anthropic states that it never reduces model quality because of demand, time of day or server load; the problems came from the bugs alone.
Context window routing error. From 5 August some Sonnet 4 requests were misrouted to servers configured for the upcoming 1M-token context window, at first 0.8% of requests. A routine load-balancing change on 29 August sent more short-context requests there, and at the worst hour, on 31 August, 16% of Sonnet 4 requests were affected. About 30% of Claude Code users who made requests in this period had at least one message routed to the wrong server type. On Amazon Bedrock misrouted traffic peaked at 0.18% of Sonnet 4 requests from 12 August, and on Vertex AI fewer than 0.0004% of requests were affected between 27 August and 16 September. The routing logic was fixed on 4 September and the fix was rolled out to the first-party platform and Vertex AI by 16 September and to Bedrock by 18 September.
Output corruption. A misconfiguration deployed to the Claude API TPU servers on 25 August caused an error during token generation: a runtime performance optimization occasionally gave high probability to tokens that should rarely appear in the context. It affected Opus 4.1 and Opus 4 from 25 to 28 August and Sonnet 4 from 25 August to 2 September, and did not affect third-party platforms. The change was rolled back on 2 September.
Approximate top-k miscompilation. On TPUs the models compute next-token probabilities in bf16, but the XLA compiler may run some operations in fp32 (the xla_allow_excess_precision flag, true by default), so operations that should agree on the most probable token sometimes disagreed and that token could drop out. In December 2024 Anthropic had deployed a workaround for the top token being dropped at temperature zero. The late-August rewrite of the sampling code fixed the precision issues and removed that workaround, which exposed a latent bug in the approximate top-k operation: for certain batch sizes and model configurations it sometimes returned completely wrong results. The bug was confirmed on Claude Haiku 3.5 and may have affected a subset of Sonnet 4 and Opus 3 on the Claude API. The change was rolled back for Haiku 3.5 on 4 September, for Opus 3 on 12 September, and for Sonnet 4 as a precaution although the bug could not be reproduced there. Anthropic switched to exact top-k, standardized some additional operations on fp32 precision, and is working with the XLA:TPU team on a compiler fix.
From early August 2025 a number of users reported that Claude's responses had become worse. At first the reports were hard to tell apart from the normal variation in user feedback. By late August they had become frequent and persistent enough for Anthropic to open an investigation.
What a user saw depended on which bug served the request. Some responses contained Thai or Chinese characters in reply to English prompts (an English question might get "สวัสดี" in the middle of the answer) or obvious syntax errors in code. Requests sent to the wrong server pool came back with degraded responses. Routing was sticky, so once a request landed on a wrong server, follow-up messages were likely to go to the same server, and some users were hit much harder than others.
The reports contradicted each other. The first bug touched about 0.8% of Sonnet 4 requests from 5 August, two more bugs came with deployments on 25 and 26 August, and a load-balancing change on 29 August started to increase the affected traffic, so many users had problems while others saw normal quality. Each bug showed different symptoms on different platforms at different rates, which looked like random, inconsistent degradation with no single cause.
The usual checks did not catch it. Benchmarks, safety evaluations, performance metrics, spot checks and canary deployments did not show what users described, partly because Claude often recovers well from an isolated mistake. Privacy controls limited engineers' access to interactions that were not sent in as feedback. When negative reports spiked on 29 August, nobody linked them right away to the routine load-balancing change. The compiler bug itself behaved inconsistently: it changed with unrelated operations before or after it and with whether debugging tools were enabled, and the same prompt could work on one request and fail on the next.
Root cause Software defect
User reports of degraded responses from early August, including reports online. They were hard to tell apart from normal feedback variation, and their rising frequency and persistence by late August led Anthropic to open an investigation; internal evaluations did not detect the degradation.
Benchmarks, safety evaluations and canary deployments missed degradation that users noticed, partly because the model often recovers from isolated mistakes, and Anthropic relied too heavily on noisy evaluations, with no clear way to connect rising reports to each of its recent changes. Overlapping bugs with different symptoms on different platforms looked like random degradation, and privacy controls on user interactions kept engineers from examining the interactions needed to identify or reproduce the bugs. Anthropic concluded that it needs evaluations running continuously on production and a continuous signal from users, such as the /bug command in Claude Code and the thumbs-down button in the Claude apps.
IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all modelsOpenAI · 2023same root cause · shared Model request routing
IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutesGoogle Cloud · 2022same root cause · shared Model request routing
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.