Graph
Cases4 cases
1 to 4 of 4
IL-0248Azure OpenAI inference requests slowed and failed in several regions as a retry storm overloaded the routing layerMicrosoft Azure2026- Symptoms
From 09:39 to 17:05 UTC on 29 May 2026, customers sending inference requests to Azure OpenAI Service saw higher latency, intermittent failures, timeouts and HTTP 5XX errors. Automated monitoring detected falling request success rates at 09:39 UTC, and regions in Europe and Australia East, which processed the most affected traffic, were hit hardest.
- Trigger
A change in an upstream API layer used by Microsoft 365 made some capacity-related failures surface as transient server failures that callers retry. It rolled out first in Australia East and later in Sweden Central.
- Root cause
- Component
- Contributing factor
- Detection
Automated monitoring detected a drop in request success rates at 09:39 UTC and started the incident investigation.
- Remediation
- Isolated the internal workload that sent the amplified retries onto dedicated infrastructure
- Rolled back the triggering change in the upstream service and reduced retry-driven traffic
- Cleared the backlogged requests
- Action items
- Move large first-party generative AI workloads onto dedicated routing infrastructure instead of the shared inference load balancing stamp
- Improve diagnostic guidance, runbooks and alerting to tell secondary crash symptoms from the real source of overload
- Work with upstream teams on retry policy design, dependency contracts and telemetry to catch retry amplification early
- Add stronger overload prevention and throttling so no single internal workload can overwhelm shared components
- Find and remove single points of failure in the inference routing layer
- Lessons
Internal workloads were not protected the way external traffic was: their traffic skipped some rate limiting and overload controls, so the amplified retries reached shared infrastructure. The apparent recovery after disabling the suspected feature hid the real cause until the problem came back at larger scale in Sweden Central.
IL-0226Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errorsMicrosoft Azure2025- Symptoms
From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.
- Trigger
At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.
- Root cause
- Component
- Contributing factor
- Detection
The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.
- Remediation
- The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
- Removed the problematic customer configurations from the latest LKG snapshot by hand
- Blocked all customer configuration changes from propagating to the data plane
- Deployed the updated LKG configuration across the global fleet
- Moved Azure Portal away from Azure Front Door
- Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
- Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place
- Action items
- Fix the control plane incompatibility and the data plane bug
- Process each customer configuration completely and synchronously before production stages
- Add stages to the phased configuration rollout, with longer bake time between them
- Move configuration processing on data plane servers into isolated worker processes
- Reduce configuration propagation time from 45 minutes to about 15 minutes
- Test backwards compatibility with configurations generated by earlier control plane builds
- Move critical first-party infrastructure to an active-active setup with fail away
- Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
- Restore customer configurations within one hour by using the local configuration cache
- Cut data plane recovery further, to about 10 minutes
- Remove delays in Azure Service Health alerts by raising resource thresholds
- Extend automated Azure Service Health alerts to similar classes of service degradation
- Fail over support request systems faster so Premier and Unified customers can always open requests
- Lessons
Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.
IL-0339Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hoursMicrosoft Azure2024- Symptoms
From 11:45 UTC on 30 July 2024, a subset of customers got intermittent connection errors, timeouts and latency spikes when connecting to Microsoft services that use Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The Azure Portal team saw the first service degradation at 11:47 UTC.
- Trigger
Around 11:45 UTC the Azure network DDoS protection service was disengaging after a routine DDoS mitigation and returning traffic to its default routing to AFD, but the routes at one site in Europe could not be updated because the network control plane could not reach that site during a local power outage.
- Root cause
- Component
- Contributing factor
- Detection
The Azure Portal team detected service degradation at 11:47 UTC and began to investigate. At 12:10 UTC network monitoring correlated the Portal incident to a network issue at one site in Europe, and networking engineers joined.
- Remediation
- Rerouted traffic under the standard congestion playbook, which removed most of the impact by 13:58 UTC
- Ordered the network control plane to reissue its commands to the misbehaving device, which could not be reached
- Removed the device that kept attracting traffic from the network and scanned the network for other issues
- Action items
- Add the missing configuration on network devices so a DDoS mitigation issue in one geography cannot spread to another
- Improve validation and monitoring in the Azure network to detect invalid configurations
- Improve monitoring for when the DDoS protection service is unreachable from the control plane but still serving traffic
IL-0225An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet lossMicrosoft Azure2023- Symptoms
From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.
- Trigger
A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.
- Root cause
- Component
- Contributing factor
- Detection
Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.
- Remediation
- Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss
- Action items
- Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
- Give on-call engineers real-time visibility of approved and unqualified activity on network devices
- Run regular mandatory operational training and attestation of following all SOPs
- Have the Change Advisory Board review all SOPs still pending qualification within 30 days
These 4 cases involve CDN and edge network and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause3
Faulty deployment1 case
- Component
- Contributing factor
- Organisation
Software defect1 case
- Contributing factor
- Organisation
Configuration change2 cases
- Component
- Contributing factor
- Organisation