Cases4 cases

1 to 4 of 4

1. IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors

   Microsoft Azure2025

   Symptoms

   From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.

   Intermittent timeoutsDNS resolution failuresService outageHigh latency

   Trigger

   At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.

   Root cause

   Software defect

   Component

   CDN and edge networkAuthoritative DNS servers and records

   Contributing factor

   Gap in automated testsChange with a delayed effectIneffective rollbackExcessive blast radius

   Detection

   The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.

   Remediation

   • The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
   • Removed the problematic customer configurations from the latest LKG snapshot by hand
   • Blocked all customer configuration changes from propagating to the data plane
   • Deployed the updated LKG configuration across the global fleet
   • Moved Azure Portal away from Azure Front Door
   • Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
   • Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place

   Action items

   • Fix the control plane incompatibility and the data plane bug
   • Process each customer configuration completely and synchronously before production stages
   • Add stages to the phased configuration rollout, with longer bake time between them
   • Move configuration processing on data plane servers into isolated worker processes
   • Reduce configuration propagation time from 45 minutes to about 15 minutes
   • Test backwards compatibility with configurations generated by earlier control plane builds
   • Move critical first-party infrastructure to an active-active setup with fail away
   • Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
   • Restore customer configurations within one hour by using the local configuration cache
   • Cut data plane recovery further, to about 10 minutes
   • Remove delays in Azure Service Health alerts by raising resource thresholds
   • Extend automated Azure Service Health alerts to similar classes of service degradation
   • Fail over support request systems faster so Premier and Unified customers can always open requests

   Lessons

   Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.

   View case
2. IL-0253 Firefox froze and could not load any web content for close to two hours, stuck in an HTTP/3 request to Telemetry

   Mozilla2022

   Symptoms

   On 13 January 2022, shortly after 07:28 UTC, Mozilla saw a spike in crashes reported through the Firefox crash reporter and received several reports, from inside and outside Mozilla, of the browser hanging. For close to two hours Firefox was unusable for users worldwide and could not load web content.

   Service outage

   Trigger

   At 07:28 UTC on 13 January 2022 GCP deployed an unannounced change that made HTTP/3 the default for load balancers set to "Automatic (default)", so some Firefox connections to Mozilla's services, including Telemetry, started using HTTP/3 instead of HTTP/2.

   Root cause

   Software defect

   Component

   Cloud load balancerHTTP and RPC client library

   Contributing factor

   Unreviewed defaultsGap in automated tests

   Detection

   A spike in crashes reported through the Firefox crash reporter, and reports of browser hangs from inside and outside Mozilla.

   Remediation

   • Explicitly disabled HTTP/3 on the GCP load balancers at 09:12 UTC

   Action items

   • Work with GCP to improve the situation around unannounced changes such as the HTTP/3 default
   • Review all service configurations to avoid similar mistakes
   • Run more system tests with different HTTP versions
   • Find ways to make the browser more resilient to such problems and to speed up incident response

   Lessons

   The hang needed three things together: a provider-side change to the HTTP version, a Rust component calling the network stack through viaduct, and a case-sensitive header lookup deep in the HTTP/3 code. Leaving the load balancers on an automatic default let the provider's change apply without any action from Mozilla, and an announcement would likely have led to a controlled test in staging first. A change of HTTP version and an extra networking layer such as viaduct are significant enough to be covered by system tests.

   View case
3. IL-0328 A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"

   CircleCI2021

   Symptoms

   From about 18:50 to 19:43 UTC on Monday, November 8, 2021, CircleCI customer jobs on all executors were blocked in a "not running" state. Errors and alerts appeared right after a deploy that changed a field type in the database of the job distribution service.

   Service outageHigh latencyScale-up failureCloud API rate limit exhausted

   Trigger

   At 18:50 UTC a deploy changed the type of a field in the PostgreSQL database of the job distribution service, in a way that was not backwards compatible.

   Root cause

   Faulty deployment

   Component

   DatabaseNode pool

   Contributing factor

   Ineffective rollbackExcessive blast radiusGap in automated tests

   Detection

   Errors and alerts related to the distribution failure appeared immediately after the 18:50 UTC deploy.

   Remediation

   • Rolled back the field type change, which did not clear the errors
   • Wrote, built and manually deployed a change that made the distribution system ignore the field
   • Manually scaled the primary compute cluster for the expected thundering herd of work
   • Monitored until processing was normal and resolved the incident

   Action items

   • Fix the gaps found in the testing strategy
   • Change how the distributor reacts to unexpected data so that this kind of deploy should not affect all jobs
   • Make recovery faster when unforeseen failures occur

   View case
4. IL-0273 A power disruption took GitHub's Redis clusters offline, the app could not boot, and GitHub served 503s for 2 hours

   GitHub2016

   Symptoms

   At 00:23 UTC on 28 January 2016 GitHub became unavailable and returned HTTP 503 responses with the unicorn error page. On-call engineers received alerts for unreachable servers and a spike in exceptions related to Redis connectivity, after slightly over 25% of the servers and several network devices in the primary data center rebooted.

   Service outage

   Trigger

   At 00:23 UTC a brief disruption in the systems that supply power to the servers and equipment of GitHub's primary data center rebooted slightly over 25% of the servers and several network devices.

   Root cause

   Dependency failure

   Component

   In-memory cacheData center power and cooling

   Contributing factor

   Unpatched known bugSingle point of failureGap in automated testsCircular dependency

   Detection

   The reboots generated alerts to several on-call engineers, starting with notifications for unreachable servers and a spike in exceptions related to Redis connectivity.

   Remediation

   • Worked with on-site facilities technicians to drain flea power from the failed servers and start them cold so their disks were visible
   • Rebuilt the affected Redis clusters on alternate hardware
   • Restored the Redis cluster data onto standby equipment and brought the Redis-server processes back online
   • Removed the maintenance page after the HAProxy load balancers returned the recovered application servers to the pool

   Action items

   • Update the server firmware with the known issue across the fleet
   • Make tooling open issues automatically when new firmware updates are available, so their changelogs are reviewed
   • Add tests that make sure application processes start even when certain external systems are unavailable
   • Improve circuit breakers so functionality degrades gracefully when backend services are down
   • Review the availability requirements of internal systems such as server provisioning so they are on par with user-facing systems
   • Strengthen communication between teams during incidents
   • Predefine escalation strategies for all-hands incidents
   • Improve status messaging to users during incidents

   Lessons

   Dependencies between components fail in new ways unless they are tested, and a power loss cannot be fully prevented, so the goal is fast and reliable recovery. The application should start without non-essential backends, and internal systems needed to recover from an outage, such as server provisioning, must be as reliable as the system being recovered.

   View case

By root cause3