Cases3 cases

1 to 3 of 3

1. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours

   Microsoft Azure2024

   Symptoms

   From 11:45 UTC on 30 July 2024, a subset of customers got intermittent connection errors, timeouts and latency spikes when connecting to Microsoft services that use Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The Azure Portal team saw the first service degradation at 11:47 UTC.

   Intermittent connection failuresIntermittent timeoutsHigh latencyPacket loss

   Trigger

   Around 11:45 UTC the Azure network DDoS protection service was disengaging after a routine DDoS mitigation and returning traffic to its default routing to AFD, but the routes at one site in Europe could not be updated because the network control plane could not reach that site during a local power outage.

   Root cause

   Configuration change

   Component

   CDN and edge networkDDoS protection serviceNetwork control plane (SDN)Data center power and cooling

   Contributing factor

   Overlapping failuresExcessive blast radiusMonitoring blind spot

   Detection

   The Azure Portal team detected service degradation at 11:47 UTC and began to investigate. At 12:10 UTC network monitoring correlated the Portal incident to a network issue at one site in Europe, and networking engineers joined.

   Remediation

   • Rerouted traffic under the standard congestion playbook, which removed most of the impact by 13:58 UTC
   • Ordered the network control plane to reissue its commands to the misbehaving device, which could not be reached
   • Removed the device that kept attracting traffic from the network and scanned the network for other issues

   Action items

   • Add the missing configuration on network devices so a DDoS mitigation issue in one geography cannot spread to another
   • Improve validation and monitoring in the Azure network to detect invalid configurations
   • Improve monitoring for when the DDoS protection service is unreachable from the control plane but still serving traffic

   View case
2. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss

   Microsoft Azure2023

   Symptoms

   From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.

   High latencyIntermittent timeoutsIntermittent connection failuresPacket lossNetwork unreachable

   Trigger

   A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.

   Root cause

   Configuration change

   Component

   Backbone network (WAN)Interior routing (IGP)BGP routing

   Contributing factor

   No configuration testsUnverified assumptionExcessive blast radiusHarmful manual action

   Detection

   Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.

   Remediation

   • Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss

   Action items

   • Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
   • Give on-call engineers real-time visibility of approved and unqualified activity on network devices
   • Run regular mandatory operational training and attestation of following all SOPs
   • Have the Change Advisory Board review all SOPs still pending qualification within 30 days

   View case
3. IL-0532 Azure Software Load Balancer sent packets of one flow to different backends, dropping Firewall and Private Link traffic

   Microsoft Azure2022

   Symptoms

   From 05:26 UTC on 28 June 2022, some Azure customers saw intermittent network failures and packet loss when connecting to resources behind a Software Load Balancer (SLB). Telemetry showed a small increase in errors and customers raised support requests, but the failures were intermittent and were not linked to a recent deployment at first.

   Intermittent connection failuresPacket lossMisrouted requests

   Trigger

   A new SLB code deployment, rolled out through Safe Deployment Practices, made some MUX instances store the backend DIPs of load balancers with more than 15 DIPs in a different order.

   Root cause

   Faulty deployment

   Component

   Cloud load balancer

   Contributing factor

   Ignored early signal

   Detection

   Telemetry showed a small increase in errors and customers raised support requests, but the health checks of Safe Deployment Practices did not correlate these intermittent signals with the deployment until the code had reached 14 regions.

   Remediation

   • Mitigated the VIPs of customers identified as impacted with targeted manual operations
   • Built and tested a hotfix and rolled it out in phases under Safe Deployment Practices
   • Ran automation that scanned all VIPs every five minutes and fixed VIPs in an inconsistent state until the hotfix was out
   • Restored consistent ordering of the backend instances to the state before the incident

   Action items

   • Add Azure Firewall monitoring that detects when the upstream and downstream packets of one flow do not reach the same DIP
   • Keep the IPv6 flow identifier consistent for the lifetime of a connection so all its packets land on one MUX
   • Improve packet capture support for firewalls
   • Add logging of TCP flows, including invalid packets and RST, FIN and FIN-ACK packets
   • Onboard more Azure services to the Gandalf AIOps solution suite for end-to-end monitoring during safe deployments

   Lessons

   Microsoft wrote that the incident showed the need for more thorough end-to-end monitoring during safe deployments.

   View case

By root cause2