Graph
Cases4 cases
1 to 4 of 4
IL-0226Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errorsMicrosoft Azure2025- Symptoms
From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.
- Trigger
At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.
- Root cause
- Component
- Contributing factor
- Detection
The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.
- Remediation
- The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
- Removed the problematic customer configurations from the latest LKG snapshot by hand
- Blocked all customer configuration changes from propagating to the data plane
- Deployed the updated LKG configuration across the global fleet
- Moved Azure Portal away from Azure Front Door
- Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
- Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place
- Action items
- Fix the control plane incompatibility and the data plane bug
- Process each customer configuration completely and synchronously before production stages
- Add stages to the phased configuration rollout, with longer bake time between them
- Move configuration processing on data plane servers into isolated worker processes
- Reduce configuration propagation time from 45 minutes to about 15 minutes
- Test backwards compatibility with configurations generated by earlier control plane builds
- Move critical first-party infrastructure to an active-active setup with fail away
- Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
- Restore customer configurations within one hour by using the local configuration cache
- Cut data plane recovery further, to about 10 minutes
- Remove delays in Azure Service Health alerts by raising resource thresholds
- Extend automated Azure Service Health alerts to similar classes of service degradation
- Fail over support request systems faster so Premier and Unified customers can always open requests
- Lessons
Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.
IL-0552CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UICircleCI2025- Symptoms
At about 23:00 UTC on 21 January 2025, an automated alert reported that the CircleCI database instance holding archived data was almost out of free storage space. From 23:50 UTC customers saw slow starts and cancellations of workflows and jobs, and delays and difficulty viewing workflows in the CircleCI UI, although pipelines for a specific project could still be viewed.
- Trigger
A code change to a function in the archival database on 16 January 2025 unintentionally made it write an excessive number of log messages, and the database filled its storage.
- Root cause
- Component
- Contributing factor
- Detection
An automated alert at about 23:00 UTC warned that the archival database was almost out of free storage, before customers were affected. The growing workflows queue was not linked to it at first because a separate alert about request volume fired at about the same time.
- Remediation
- Scaled up the event consumer that processes the workflows queue at 23:44 UTC
- Promoted a read replica of the archival database to a standalone primary at 23:55 UTC
- Action items
- Fix the database function so it no longer writes an excessive number of log messages
- Add a max duration to the workflows retry policy for archiving so it fails earlier than the default timeout
- Move workflow archival to an event-based model that decouples it from the workflows service
- Add forecast and anomaly monitoring for databases that alerts on unusual activity before it reaches critical levels
- Lessons
The storage alert on the archival database did fire, but the database filled much faster than forecast, so there was little time to act before customers were affected. Because the workflows service called the archival service synchronously, a problem in the archival database slowed the whole workflows service.
IL-0314Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restartingDaily2023- Symptoms
At 9:47 AM Eastern (14:47 UTC) on 7 February 2023, Daily's database alerts reported a performance problem under normal operational load: a high level of lock contention on a database that had been upgraded the weekend before and had run normally since Monday. Users could get websocket connection errors when joining calls, and REST API requests could time out or return 5xx errors.
- Trigger
Database maintenance the weekend before had upgraded the database to a MySQL point release with a low-level behaviour change, and on 7 February, under normal load, that change produced the deadlocks Daily saw on Aurora MySQL 2.11.0.
- Root cause
- Component
- Contributing factor
- Detection
Database alerts at 9:47 AM Eastern reported a performance issue with high lock contention on the newly upgraded database.
- Remediation
- Disabled MySQL binlog replication to the old database and restarted the database to reduce load
- The database restarted by itself at 10:47 AM, the SFUs dropped the stale meeting updates without restarting, and the platform recovered
- Rate-limited the POST /rooms endpoint to the same limit as DELETE /rooms/:name
- Updated an index on the affected table in production on the Saturday evening to work around the MySQL behaviour change
- Action items
- Work with AWS so the Aurora MySQL 2.11.0 deadlock issue is documented, understood and fixed in a future release
- Add more API request patterns to the testing workload used for infrastructure changes
- Add production monitoring alarms on more fine-grained database metrics
- Lessons
Daily tests infrastructure changes under production-like workloads, but its synthetic workload did not have the shape that triggers these deadlocks. A deadlock behaviour change in MySQL 5.7 that was already known turned out far more severe in production than either Daily or the AWS Aurora team expected.
IL-0241A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe2019- Symptoms
On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.
- Trigger
The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.
- Root cause
- Component
- Contributing factor
- Detection
Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.
- Remediation
- Restarted all nodes of the database cluster, which forced a successful primary election
- Updated the production configuration and restarted the database cluster's nodes
- Ramped traffic back up, putting services for user-initiated API requests first
- Action items
- Alert when nodes stop reporting their replication lag
- Alert when a shard enters a state that could trigger the election fault
- Work with the database maintainers on a fix for the underlying fault
- Add circuit breaking on failed operations to particular database clusters
- Use fault isolation to contain the impact of a single failed shard
- Limit the resources used by clients that keep retrying failed requests
- Add procedures and tooling that make rapid configuration changes during incident response safer
These 4 cases involve Database and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause3
Software defect2 cases
- Component
- Contributing factor
- Organisation
Faulty deployment1 case
- Contributing factor
- Organisation
Breaking change in an upgrade1 case
- Component
- Contributing factor
- Organisation