Cases3 cases

1 to 3 of 3

1. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours

   Microsoft Azure2024

   Symptoms

   From 11:45 UTC on 30 July 2024, a subset of customers got intermittent connection errors, timeouts and latency spikes when connecting to Microsoft services that use Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The Azure Portal team saw the first service degradation at 11:47 UTC.

   Intermittent connection failuresIntermittent timeoutsHigh latencyPacket loss

   Trigger

   Around 11:45 UTC the Azure network DDoS protection service was disengaging after a routine DDoS mitigation and returning traffic to its default routing to AFD, but the routes at one site in Europe could not be updated because the network control plane could not reach that site during a local power outage.

   Root cause

   Configuration change

   Component

   CDN and edge networkDDoS protection serviceNetwork control plane (SDN)Data center power and cooling

   Contributing factor

   Overlapping failuresExcessive blast radiusMonitoring blind spot

   Detection

   The Azure Portal team detected service degradation at 11:47 UTC and began to investigate. At 12:10 UTC network monitoring correlated the Portal incident to a network issue at one site in Europe, and networking engineers joined.

   Remediation

   • Rerouted traffic under the standard congestion playbook, which removed most of the impact by 13:58 UTC
   • Ordered the network control plane to reissue its commands to the misbehaving device, which could not be reached
   • Removed the device that kept attracting traffic from the network and scanned the network for other issues

   Action items

   • Add the missing configuration on network devices so a DDoS mitigation issue in one geography cannot spread to another
   • Improve validation and monitoring in the Azure network to detect invalid configurations
   • Improve monitoring for when the DDoS protection service is unreachable from the control plane but still serving traffic

   View case
2. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled

   Google Cloud2019

   Symptoms

   On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.

   Packet lossNetwork unreachableHigh latencyHigh error rateMonitoring outage

   Trigger

   At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingBackbone network (WAN)

   Contributing factor

   Excessive blast radiusCircular dependencyLimited debug accessIneffective rollback

   Detection

   Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.

   Remediation

   • Lowered the priority of less critical network traffic classes to reduce congestion
   • Halted the datacenter automation software responsible for the maintenance event
   • Rescheduled the network control plane and its supporting infrastructure in each location
   • Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
   • Directed service traffic away from the affected regions
   • Moved the Cloud VPN control plane in three regions out of its fail-safe state

   Action items

   • Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
   • Harden the cluster management software to reject such requests whatever their origin
   • Reconfigure the network control plane to reject maintenance requests of the type involved
   • Make the network control plane in each location persist its configuration
   • Let the network stay in fail static mode longer when the control plane is lost
   • Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
   • Extend continuous disaster recovery testing to this and similarly catastrophic failures

   View case
3. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic

   Packet2019

   Symptoms

   At about 08:49 UTC on 2 June 2019, Packet's internal network monitoring alerted on serious reachability and performance problems in several of Packet's global datacenter locations. The first investigation found routing loops and traffic being blackholed to specific internal and Internet destinations.

   Network unreachablePacket lossHigh latency

   Trigger

   A trans-continental US backbone link in the network of Packet's upstream provider Zayo had a hardware failure and blackholed IP traffic routed over it.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingNetwork control plane (SDN)

   Contributing factor

   –

   Detection

   Packet's internal network monitoring alerted at 08:49 UTC on reachability issues in several global datacenter locations, and engineers confirmed them at 08:52.

   Remediation

   • Migrated traffic to an alternative network and removed Zayo from the provider mix
   • Applied route filters to return service to normal
   • Optimised specific routing policies to improve latency across the backbone with the new traffic mix

   Action items

   • Expand internal connectivity monitoring and alerting of on-call operations teams
   • Expand network automation and routing so traffic can be balanced across all available backbone paths and an operator can trigger automated re-routes across all Packet datacenters in a major provider incident
   • Make Packet's provisioning systems less dependent on third party network providers and inter-datacenter connectivity

   View case

By root cause3