Incident Library

Applied filters Root causeSoftware defectComponentCDN and edge networkSymptomHigh latencyClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases3 cases

1 to 3 of 3

IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors
Microsoft Azure2025
Symptoms
From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.
Intermittent timeouts DNS resolution failures Service outage High latency
Trigger
At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.
Root cause
Software defect
Component
CDN and edge network Authoritative DNS servers and records
Contributing factor
Gap in automated tests Change with a delayed effect Ineffective rollback Excessive blast radius
Detection
The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.
Remediation
The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
Removed the problematic customer configurations from the latest LKG snapshot by hand
Blocked all customer configuration changes from propagating to the data plane
Deployed the updated LKG configuration across the global fleet
Moved Azure Portal away from Azure Front Door
Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place
Action items
Fix the control plane incompatibility and the data plane bug
Process each customer configuration completely and synchronously before production stages
Add stages to the phased configuration rollout, with longer bake time between them
Move configuration processing on data plane servers into isolated worker processes
Reduce configuration propagation time from 45 minutes to about 15 minutes
Test backwards compatibility with configurations generated by earlier control plane builds
Move critical first-party infrastructure to an active-active setup with fail away
Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
Restore customer configurations within one hour by using the local configuration cache
Cut data plane recovery further, to about 10 minutes
Remove delays in Azure Service Health alerts by raising resource thresholds
Extend automated Azure Service Health alerts to similar classes of service degradation
Fail over support request systems faster so Premier and Unified customers can always open requests
Lessons
Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.
View case
IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests
Google Cloud2018
Symptoms
From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.
High error rate Connection resets High latency
Trigger
A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.
Root cause
Software defect
Component
Cloud load balancer CDN and edge network
Contributing factor
Gap in automated tests Excessive blast radius
Detection
Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.
Remediation
Reverted the configuration change at 12:44 PDT, which stopped the GFE restarts
Action items
Add safeguards that disable features not yet in service
Harden the GFE testing stack against latent bugs in production binaries that make a task restart
Isolate the shards of GFE pools further from each other to reduce the scope of failures
Build a consolidated dashboard of all configuration changes for GFE pools
View case
IL-0388 A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guilds
Discord2017
Symptoms
At about 14:01 PDT on 13 October 2017, Discord engineers were alerted to anomalies by real-time metrics and monitoring. A Redis instance that was the primary of a highly available cluster used by the API services had dropped offline after Google Cloud Platform migrated it automatically, and some API instances did not handle the failover to the secondary node correctly.
Failover failure High latency Service outage
Trigger
Google Cloud Platform automatically migrated the Redis instance that was the primary of a highly available cluster used by the API services, and the node incorrectly dropped offline.
Root cause
Software defect
Component
In-memory cache Database CDN and edge network
Contributing factor
Unpatched known bug Overlapping failures Monitoring blind spot
Detection
Real-time metrics and monitoring alerted engineers at 14:01 PDT. The second phase was seen at 15:41 as another anomaly in monitoring, and end user reports of servers appearing offline were escalated at 15:42.
Remediation
Ran a rolling restart of all API instances to clear the Redis connection issues
Corrected the misconfigured edge caching rule for the expensive API route
Disabled the problem API endpoint on the edge network
Recovered the Cassandra cluster
Ran a second rolling restart of all API instances for the latency left by the Cassandra issues
Removed the misbehaving guilds node from the service hashring
Forcefully restarted the misbehaving guilds node
Restarted the whole service, forcing all users to reconnect
Blocked traffic that relied on the failing push component to reduce its load
Corrected the API setting that lets users reconnect quickly and safely
Action items
Fix the known issues with Redis failover within the coming weeks
Change the cached route so this type of error does not degrade other services
Monitor the signals that would have warned of the cascading failure
Improve the reliability and failure handling of the affected services
Roll out an update of the library with the suspected flaws
View case

Check your cluster for this

These 3 cases involve CDN and edge network and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

Graph

Cases3 cases

By root cause1

Software defect3 cases