Cases7 cases

4 to 7 of 7

1. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed

   DigitalOcean2023

   Symptoms

   From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

   Network unreachableIntermittent timeoutsHigh latency

   Trigger

   The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

   Root cause

   Software defect

   Component

   BGP routing

   Contributing factor

   Excessive blast radiusHarmful manual action

   Remediation

   • Rolled out a software update to the edge routers running the affected version, site by site
   • Rebalanced traffic on the congested routes between AMS3 and FRA1
   • Rolled out the fix to the redundant networking equipment so a failover would not hit the bug

   Action items

   • Upgrade all global edge routers to remove the bug
   • Bring the network upgrades already done in Asia to Europe and North America

   View case
2. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed

   Linode2023

   Symptoms

   On 12 February 2023, customers in Linode's US-West region could not reach their Linodes. Santa Clara customers lost Internet connectivity after a change made at 8:45 UTC until it was reverted at 10:13 UTC, and later in the day the connection between the Fremont and Santa Clara data centers was cut completely, leaving both with degraded service or a full outage until 16:25 UTC; customers could also have trouble provisioning, powering down or rebooting services.

   Network unreachable

   Trigger

   A single link of the A-side fibre circuit between Fremont and Santa Clara went down on 11 February, the whole B-side circuit followed on 12 February, and during troubleshooting with the hardware vendor a non-intrusive command triggered a rare bug that took down all A-side links.

   Root cause

   Software defect

   Component

   Backbone network (WAN)NIC, cable and switch port link

   Contributing factor

   Overlapping failuresHarmful manual action

   Detection

   The response team began investigating when the A-side link went down on 11 February. The status page opened at 10:22 UTC on 12 February to investigate reports of connectivity issues in the US-West region.

   Remediation

   • Reverted the change that removed internet traffic from the Santa Clara border routers
   • Performed an emergency code upgrade of both circuits, restoring connectivity between the sites
   • Restored the B-side path with the hardware vendor and fiber provider, regaining full redundancy

   View case
3. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled

   Google Cloud2019

   Symptoms

   On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.

   Packet lossNetwork unreachableHigh latencyHigh error rateMonitoring outage

   Trigger

   At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingBackbone network (WAN)

   Contributing factor

   Excessive blast radiusCircular dependencyLimited debug accessIneffective rollback

   Detection

   Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.

   Remediation

   • Lowered the priority of less critical network traffic classes to reduce congestion
   • Halted the datacenter automation software responsible for the maintenance event
   • Rescheduled the network control plane and its supporting infrastructure in each location
   • Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
   • Directed service traffic away from the affected regions
   • Moved the Cloud VPN control plane in three regions out of its fail-safe state

   Action items

   • Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
   • Harden the cluster management software to reject such requests whatever their origin
   • Reconfigure the network control plane to reject maintenance requests of the type involved
   • Make the network control plane in each location persist its configuration
   • Let the network stay in fail static mode longer when the control plane is lost
   • Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
   • Extend continuous disaster recovery testing to this and similarly catastrophic failures

   View case
4. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h

   Google Cloud2017

   Symptoms

   On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.

   Network unreachableProbe failures

   Trigger

   A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.

   Root cause

   Software defect

   Component

   Cloud load balancerNetwork control plane (SDN)

   Contributing factor

   Excessive blast radiusGap in automated tests

   Remediation

   • Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
   • Applied the fix zone by zone between 11:36 and 12:42

   Action items

   • Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
   • Improve the inefficient code path
   • Write new tests that check behaviour on a wider range of configurations
   • Accelerate replacing global propagation of address configuration with decentralized routing
   • Add metrics and alerting that identify this kind of issue sooner

   View case

By root cause1