Graph
Cases3 cases
1 to 3 of 3
IL-0226Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errorsMicrosoft Azure2025- Symptoms
From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.
- Trigger
At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.
- Root cause
- Component
- Contributing factor
- Detection
The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.
- Remediation
- The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
- Removed the problematic customer configurations from the latest LKG snapshot by hand
- Blocked all customer configuration changes from propagating to the data plane
- Deployed the updated LKG configuration across the global fleet
- Moved Azure Portal away from Azure Front Door
- Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
- Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place
- Action items
- Fix the control plane incompatibility and the data plane bug
- Process each customer configuration completely and synchronously before production stages
- Add stages to the phased configuration rollout, with longer bake time between them
- Move configuration processing on data plane servers into isolated worker processes
- Reduce configuration propagation time from 45 minutes to about 15 minutes
- Test backwards compatibility with configurations generated by earlier control plane builds
- Move critical first-party infrastructure to an active-active setup with fail away
- Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
- Restore customer configurations within one hour by using the local configuration cache
- Cut data plane recovery further, to about 10 minutes
- Remove delays in Azure Service Health alerts by raising resource thresholds
- Extend automated Azure Service Health alerts to similar classes of service degradation
- Fail over support request systems faster so Premier and Unified customers can always open requests
- Lessons
Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.
IL-0250slack.com stopped resolving on some DNSSEC-validating resolvers for about 24 hours after DNSSEC signing was turned onSlack2021- Symptoms
Soon after Slack published the DS record for slack.com at its DNS registrar on 30 September 2021, the Customer Experience team started getting user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients. Fewer than 1% of online users were affected, and most reports came from users of one private corporate recursive resolver and of Google Public DNS (8.8.8.8).
- Trigger
On 30 September 2021 the Traffic Engineering team enabled DNSSEC signing on the slack.com zone in Amazon Route 53 and on its sub-delegated zones in NS1, and after a three-hour soak published the slack.com DS record at the DNS registrar.
- Root cause
- Component
- Contributing factor
- Detection
Customer Experience received user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients, and customers sent NetLog captures through support cases. After signing was rolled back, Slack's external DNS and DNSSEC resolution probes alerted the Traffic team.
- Remediation
- Removed the slack.com DS record from the DNS registrar
- Rolled back DNSSEC signing on the slack.com authoritative and delegated zones, which made resolution worse
- Asked major ISPs and public DNS resolver operators to flush their cached slack.com records
- Flushed the slack.com records on Google's 8.8.8.8 and Cloudflare's 1.1.1.1 through their public flush sites
- Lessons
The DNSSEC rollout had been tested only on domains without a wildcard record, so the Route 53 wildcard NSEC bug first appeared on slack.com; Slack names not testing a wildcard domain first as an oversight. Removing the DS record at the registrar does not stop validation at once, because resolvers cache the DS record from the .com zone for up to 24 hours, and removing the DNSKEY while that record is still cached makes every validating resolver fail. Managing DNS through Terraform hid the console warning about disabling DNSSEC. Re-enabling signing would neither have helped nor hurt, because Route 53 generates a new ZSK each time signing is enabled.
IL-0229bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDNbunny.net2021- Symptoms
On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.
- Trigger
At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.
- Root cause
- Component
- Contributing factor
- Detection
About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.
- Remediation
- Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
- Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
- Migrated the deployment systems and files to a third-party cloud storage service
- Rewired the deployment system and DNS software to the new storage and deployed
- Synced the databases and deployed the newest file sets through the third-party service
- Action items
- Phase out the BinaryPack library as a hot-fix
- Test third-party libraries more extensively
- Migrate the internal APIs to an independent third-party service
- Find how to stop non-critical software from becoming a single point of failure across clusters
- Make the DNS system keep a local copy of all backup data with automatic failure detection
- Lessons
bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.
These 3 cases involve Authoritative DNS servers and records and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause2
Software defect2 cases
- Component
- Contributing factor
- Organisation
Faulty deployment1 case
- Contributing factor
- Organisation