IL-0339 Microsoft Azure · 2024-07-30azurenetworking
Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours
Azure Front Door is Microsoft's platform for web acceleration, global load balancing and content delivery, running in nearly 200 locations. AFD and Azure CDN have their own DDoS defences and also rely on the Azure network DDoS protection service for network-layer attacks. Between 10:15 and 10:45 UTC on 30 July 2024, a TCP SYN flood hit several AFD and CDN sites and was mitigated automatically. At about 11:45 UTC, as the DDoS protection service disengaged and default routing to AFD resumed, the network control plane could not reach one site in Europe because of a local power outage, so routes there could not be updated and traffic inside Europe kept flowing to AFD through the DDoS protection service. On its own this would not have hurt customers, and it was unrelated to the attack.
A separate latent network configuration issue, however, routed traffic from outside Europe to the DDoS protection system inside Europe. The extra traffic congested that site and caused high latency and connection failures across several regions. Engineers confirmed the congestion at 12:55 UTC and rerouted traffic, which removed most of the impact by 13:58 UTC. The remaining isolated failures came from a device in Europe that did not obey the network control plane and kept attracting traffic; because the device was not reachable, engineers began removing it from the network safely at 17:50 UTC, finished at 19:32 UTC, and confirmed availability back at pre-incident levels at 19:43 UTC.
- Severity
- –
- Scope
- Microsoft services that use Azure Front Door and Azure CDN, and downstream services including the Azure portal and a subset of Microsoft 365 and Microsoft Purview services
- Users affected
- a subset of customers; from 13:58 to 19:43 UTC a smaller set
- Region
- multiple regions; congestion at one site in Europe
- Time to detect
- 2m
- Time to resolve
- 7h 58m
Symptoms
From 11:45 UTC on 30 July 2024, a subset of customers got intermittent connection errors, timeouts and latency spikes when connecting to Microsoft services that use Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The Azure Portal team saw the first service degradation at 11:47 UTC.
The trouble reached services downstream of AFD and CDN, including the Azure portal and a subset of Microsoft 365 and Microsoft Purview services, and customers in multiple regions saw high latency and connectivity failures. At 12:10 UTC Microsoft's network monitoring tied the Portal incident to a network issue at one site in Europe, and at 12:55 UTC engineers confirmed local congestion there, with packet loss for AFD frontends.
Rerouting traffic under the standard congestion playbook removed most of the impact by 13:58 UTC, but a smaller set of customers kept seeing a low rate of connection timeouts until 19:43 UTC. While chasing these isolated failures, engineers found at 16:15 UTC a device in Europe that kept attracting traffic after the network control plane had told it to stop. Asking the control plane to reissue its commands at 16:58 UTC did not help, because the device could not be reached.
Earlier that day, between 10:15 and 10:45 UTC, a volumetric TCP SYN flood DDoS attack hit several AFD and CDN sites and was mitigated automatically with little customer impact. During that mitigation the DDoS protection sent SYN authentication challenges, and a small subset of customers whose applications had no retry logic may have seen connection failures.
Symptom Intermittent connection failuresIntermittent timeoutsHigh latencyPacket loss
Root cause
Root cause Configuration change
- Trigger
- Around 11:45 UTC the Azure network DDoS protection service was disengaging after a routine DDoS mitigation and returning traffic to its default routing to AFD, but the routes at one site in Europe could not be updated because the network control plane could not reach that site during a local power outage.
Detection
The Azure Portal team detected service degradation at 11:47 UTC and began to investigate. At 12:10 UTC network monitoring correlated the Portal incident to a network issue at one site in Europe, and networking engineers joined.
Remediation
- Rerouted traffic under the standard congestion playbook, which removed most of the impact by 13:58 UTC
- Ordered the network control plane to reissue its commands to the misbehaving device, which could not be reached
- Removed the device that kept attracting traffic from the network and scanned the network for other issues
Action items
- Add the missing configuration on network devices so a DDoS mitigation issue in one geography cannot spread to another
- Improve validation and monitoring in the Azure network to detect invalid configurations
- Improve monitoring for when the DDoS protection service is unreachable from the control plane but still serving traffic
Lessons
–
Sources
Related cases
IL-0512xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedMIT Open Learning · 2026 · sev1same root cause · shared CDN and edge network
IL-0283Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature fileCloudflare · 2025same root cause · shared CDN and edge network
IL-0363static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project · 2023same root cause · shared CDN and edge network
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0339
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0339/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.