IL-0226 Microsoft Azure · 2025-10-29azurenetworking

Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors

The AFD control plane generates customer configuration metadata that the data plane on globally distributed edge sites consumes, and every customer change is deployed to all edge sites. The incompatible metadata exposed a latent bug in the data plane, which crashed during asynchronous processing. The defect was not caught because pre-production validation does not test all features across different control plane build versions. AFD's configuration protection system advances a change stage by stage, waiting about a minute between stages and moving on only after healthy signals from the data plane, and after a successful rollout it updates a Last Known Good (LKG) snapshot of healthy configurations for automatic rollback.

The change reached a pre-production stage at 15:36 UTC and the health signals stayed good, because the crash came asynchronously about five minutes later. By then the configuration had passed the safeguards, reached most edge sites by 15:39 UTC, and been written into the LKG snapshot. When the crashes started, the protection system blocked all new and in-flight configuration changes at 15:43 UTC. Because the latest LKG held the bad metadata, the team did not revert to it or to older LKG versions; from 17:10 UTC it removed the problematic customer configurations from the latest LKG by hand, blocked all customer configuration propagation at 17:30 UTC and began deploying the edited LKG at 17:40 UTC. Every edge site had to reload all customer configurations and traffic was rebalanced gradually to avoid overload: AFD DNS servers recovered at 18:30 UTC and traffic was moved by hand to a few healthy edge sites, automatic traffic management resumed at 20:20 UTC, and impact was mitigated at 00:05 UTC on 30 October.

After mitigation, AFD customer configuration changes were blocked at the Azure Resource Manager level until new safeguards were in place, and the block was lifted on 5 November 2025. Microsoft states that this incident was not directly related to the AFD incident of 9 October (QNBQ-5W8): the failure mode was similar but the defects were different.

Severity: –
Scope: Azure Front Door and Azure CDN and the Azure and Microsoft services behind them, including Azure Portal, App Service, SQL Database, Microsoft 365, Entra ID, Dynamics 365, Power Platform and support case creation
Users affected: –
Region: global
Time to detect: 7m
Time to resolve: 8h 24m

Symptoms

From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.

AFD edge servers crashed as they processed an incompatible customer configuration, first in the pre-production environment at 15:41 UTC and on all edge sites worldwide by 15:45 UTC. AFD's internal DNS service is hosted on the edge sites, so a subset of AFD customer requests also got intermittent DNS resolution errors.

Configuration propagation monitoring had kept receiving healthy signals after the problematic metadata went in, because the data plane crash surfaced asynchronously about five minutes later; by then the configuration had passed the protection safeguards and reached most edge sites by 15:39 UTC. From 16:15 UTC the investigation focused on AFD configuration changes.

The affected services included Azure Portal, Azure App Service, Azure SQL Database, Azure Active Directory B2C, Azure Databricks, Microsoft 365, Microsoft Entra ID, Dynamics 365 and Power Platform, and customers could not open support cases in the Azure Portal or by phone. Some services, such as the Entra and Intune portals, failed over away from AFD; Azure Portal moved away from AFD at 17:26 UTC, but parts of it without a fallback, such as Marketplace, kept failing.

From 18:30 UTC availability improved gradually, with higher latency, and availability and latency were back at pre-incident levels at 00:05 UTC on 30 October.

The first communication went to the public status page at 16:18 UTC and targeted communications to impacted customers to Azure Service Health at 16:20 UTC. Alerts delivered through Azure Service Health to impacted customers were delayed.

Symptom Intermittent timeouts DNS resolution failures Service outage High latency

Root cause

Root cause Software defect

Component CDN and edge network Authoritative DNS servers and records

Contributing factor Gap in automated tests Change with a delayed effect Ineffective rollback Excessive blast radius

Trigger: At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.

Detection

The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.

Remediation

The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
Removed the problematic customer configurations from the latest LKG snapshot by hand
Blocked all customer configuration changes from propagating to the data plane
Deployed the updated LKG configuration across the global fleet
Moved Azure Portal away from Azure Front Door
Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place

Action items

Fix the control plane incompatibility and the data plane bug
Process each customer configuration completely and synchronously before production stages
Add stages to the phased configuration rollout, with longer bake time between them
Move configuration processing on data plane servers into isolated worker processes
Reduce configuration propagation time from 45 minutes to about 15 minutes
Test backwards compatibility with configurations generated by earlier control plane builds
Move critical first-party infrastructure to an active-active setup with fail away
Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
Restore customer configurations within one hour by using the local configuration cache
Cut data plane recovery further, to about 10 minutes
Remove delays in Azure Service Health alerts by raising resource thresholds
Extend automated Azure Service Health alerts to similar classes of service degradation
Fail over support request systems faster so Premier and Unified customers can always open requests

Lessons

Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.

Sources

https://azure.status.microsoft/en-us/status/history/?trackingId=YKYN-BWZ · official-status · archived copy

References

Related cases

IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025
same root cause · shared Authoritative DNS servers and records
IL-0250 slack.com stopped resolving on some DNSSEC-validating resolvers for about 24 hours after DNSSEC signing was turned onSlack · 2021
same root cause · shared Authoritative DNS servers and records
IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addressesGandi · 2020
same root cause · shared Authoritative DNS servers and records
IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requestsGoogle Cloud · 2018
same root cause · shared CDN and edge network
IL-0388 A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017
same root cause · shared CDN and edge network

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CDN and edge network, Authoritative DNS servers and records among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0226
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0226/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.