IL-0248 Microsoft Azure · 2026-05-29azureai-infra
Azure OpenAI inference requests slowed and failed in several regions as a retry storm overloaded the routing layer
Azure OpenAI uses an internal routing and load balancing layer to send inference requests across regions and model capacity. Microsoft 365 uses Azure OpenAI for some requests, including Copilot experiences. A change in an upstream API layer used by that workload altered how capacity-related failures were passed up the request path: lower-level capacity or resource exhaustion conditions that used to end in a response that did not set off broad retries now surfaced as transient server failures. Several layers of the calling stack treated them as retriable and retried immediately and repeatedly without enough backoff or jitter, up to 48 extra attempts for a single failed request, so internal traffic rose sharply although customer demand had not changed much. This internal traffic was not subject to the rate limiting and overload controls used for many external callers, and part of it reached the routing layer more directly than external traffic.
The change rolled out first in Australia East, where the retries began exhausting the routing layer at 09:20 UTC. Disabling an internal error-handling feature at 12:17 UTC seemed to work because Australia East traffic fell at the same time. When the rollout expanded to Sweden Central at 14:20 UTC, its larger traffic made the retry storm overload the shared routing path again, with resource exhaustion and out-of-memory crashes. At 16:27 UTC engineers confirmed that the amplified traffic came from an internal first-party workload. They isolated that workload onto dedicated infrastructure, the upstream service began rolling back the triggering change at 16:25 UTC and reduced retry traffic, and after the backlogged requests were cleared the customer impact was declared mitigated at 17:05 UTC.
- Severity
- –
- Scope
- Azure OpenAI Service inference requests: higher latency, intermittent failures, timeouts and HTTP 5XX errors
- Users affected
- –
- Region
- Multiple regions, most in Europe and Australia East (Australia East and Sweden Central named)
- Time to detect
- 19m
- Time to resolve
- 7h 45m
Symptoms
From 09:39 to 17:05 UTC on 29 May 2026, customers sending inference requests to Azure OpenAI Service saw higher latency, intermittent failures, timeouts and HTTP 5XX errors. Automated monitoring detected falling request success rates at 09:39 UTC, and regions in Europe and Australia East, which processed the most affected traffic, were hit hardest.
The failure came in two waves. The first began at 09:20 UTC as an upstream change rolled out in Australia East, and by 10:10 UTC instability and correlated failures were seen across regions and models. Early crash signatures pointed to an internal feature related to error handling, and engineers disabled it at 12:17 UTC. Service health improved from 12:40 UTC, but only because traffic in Australia East declined as part of its normal regional usage pattern, so the mitigation looked effective although the feature was not the cause.
When the upstream rollout reached Sweden Central at 14:20 UTC, a major European processing hub with much more traffic, the problem came back at 14:40 UTC at greater scale. The inference routing layer ran out of resources and multiple instances crashed with out-of-memory errors. The affected component scaled out, but not enough, and in a few regions scale-out was limited by the compute quota available for replacement virtual machine SKUs. At 16:20 UTC engineers concluded that the earlier mitigation had addressed a secondary symptom and turned to the upstream retry behaviour.
Symptom High latencyHigh error rateIntermittent timeoutsOOMKilled
Root cause
Root cause Faulty deployment
- Trigger
- A change in an upstream API layer used by Microsoft 365 made some capacity-related failures surface as transient server failures that callers retry. It rolled out first in Australia East and later in Sweden Central.
Detection
Automated monitoring detected a drop in request success rates at 09:39 UTC and started the incident investigation.
Remediation
- Isolated the internal workload that sent the amplified retries onto dedicated infrastructure
- Rolled back the triggering change in the upstream service and reduced retry-driven traffic
- Cleared the backlogged requests
Action items
- Move large first-party generative AI workloads onto dedicated routing infrastructure instead of the shared inference load balancing stamp
- Improve diagnostic guidance, runbooks and alerting to tell secondary crash symptoms from the real source of overload
- Work with upstream teams on retry policy design, dependency contracts and telemetry to catch retry amplification early
- Add stronger overload prevention and throttling so no single internal workload can overwhelm shared components
- Find and remove single points of failure in the inference routing layer
Lessons
Internal workloads were not protected the way external traffic was: their traffic skipped some rate limiting and overload controls, so the amplified retries reached shared infrastructure. The apparent recovery after disabling the suspected feature hid the real cause until the problem came back at larger scale in Sweden Central.
Sources
- https://azure.status.microsoft/en-us/status/history/?trackingId=LYXT-C1Z · official-status
References
Related cases
–
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0248
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0248/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.