Cases2 cases

1 to 2 of 2

1. IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors

   Microsoft Azure2025

   Symptoms

   From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.

   Intermittent timeoutsDNS resolution failuresService outageHigh latency

   Trigger

   At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.

   Root cause

   Software defect

   Component

   CDN and edge networkAuthoritative DNS servers and records

   Contributing factor

   Gap in automated testsChange with a delayed effectIneffective rollbackExcessive blast radius

   Detection

   The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.

   Remediation

   • The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
   • Removed the problematic customer configurations from the latest LKG snapshot by hand
   • Blocked all customer configuration changes from propagating to the data plane
   • Deployed the updated LKG configuration across the global fleet
   • Moved Azure Portal away from Azure Front Door
   • Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
   • Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place

   Action items

   • Fix the control plane incompatibility and the data plane bug
   • Process each customer configuration completely and synchronously before production stages
   • Add stages to the phased configuration rollout, with longer bake time between them
   • Move configuration processing on data plane servers into isolated worker processes
   • Reduce configuration propagation time from 45 minutes to about 15 minutes
   • Test backwards compatibility with configurations generated by earlier control plane builds
   • Move critical first-party infrastructure to an active-active setup with fail away
   • Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
   • Restore customer configurations within one hour by using the local configuration cache
   • Cut data plane recovery further, to about 10 minutes
   • Remove delays in Azure Service Health alerts by raising resource thresholds
   • Extend automated Azure Service Health alerts to similar classes of service degradation
   • Fail over support request systems faster so Premier and Unified customers can always open requests

   Lessons

   Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.

   View case
2. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case

By root cause2