Cases3 cases

1 to 3 of 3

1. IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors

   Microsoft Azure2025

   Symptoms

   From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.

   Intermittent timeoutsDNS resolution failuresService outageHigh latency

   Trigger

   At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.

   Root cause

   Software defect

   Component

   CDN and edge networkAuthoritative DNS servers and records

   Contributing factor

   Gap in automated testsChange with a delayed effectIneffective rollbackExcessive blast radius

   Detection

   The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.

   Remediation

   • The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
   • Removed the problematic customer configurations from the latest LKG snapshot by hand
   • Blocked all customer configuration changes from propagating to the data plane
   • Deployed the updated LKG configuration across the global fleet
   • Moved Azure Portal away from Azure Front Door
   • Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
   • Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place

   Action items

   • Fix the control plane incompatibility and the data plane bug
   • Process each customer configuration completely and synchronously before production stages
   • Add stages to the phased configuration rollout, with longer bake time between them
   • Move configuration processing on data plane servers into isolated worker processes
   • Reduce configuration propagation time from 45 minutes to about 15 minutes
   • Test backwards compatibility with configurations generated by earlier control plane builds
   • Move critical first-party infrastructure to an active-active setup with fail away
   • Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
   • Restore customer configurations within one hour by using the local configuration cache
   • Cut data plane recovery further, to about 10 minutes
   • Remove delays in Azure Service Health alerts by raising resource thresholds
   • Extend automated Azure Service Health alerts to similar classes of service degradation
   • Fail over support request systems faster so Premier and Unified customers can always open requests

   Lessons

   Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.

   View case
2. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed

   DigitalOcean2023

   Symptoms

   From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

   Network unreachableIntermittent timeoutsHigh latency

   Trigger

   The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

   Root cause

   Software defect

   Component

   BGP routing

   Contributing factor

   Excessive blast radiusHarmful manual action

   Remediation

   • Rolled out a software update to the edge routers running the affected version, site by site
   • Rebalanced traffic on the congested routes between AMS3 and FRA1
   • Rolled out the fix to the redundant networking equipment so a failover would not hit the bug

   Action items

   • Upgrade all global edge routers to remove the bug
   • Bring the network upgrades already done in Asia to Europe and North America

   View case
3. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

   Stripe2019

   Symptoms

   On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

   High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

   Trigger

   The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures

   Detection

   Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

   Remediation

   • Restarted all nodes of the database cluster, which forced a successful primary election
   • Updated the production configuration and restarted the database cluster's nodes
   • Ramped traffic back up, putting services for user-initiated API requests first

   Action items

   • Alert when nodes stop reporting their replication lag
   • Alert when a shard enters a state that could trigger the election fault
   • Work with the database maintainers on a fix for the underlying fault
   • Add circuit breaking on failed operations to particular database clusters
   • Use fault isolation to contain the impact of a single failed shard
   • Limit the resources used by clients that keep retrying failed requests
   • Add procedures and tooling that make rapid configuration changes during incident response safer

   View case

By root cause1