Cases2 cases

1 to 2 of 2

1. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed

   DigitalOcean2023

   Symptoms

   From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

   Network unreachableIntermittent timeoutsHigh latency

   Trigger

   The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

   Root cause

   Software defect

   Component

   BGP routing

   Contributing factor

   Excessive blast radiusHarmful manual action

   Remediation

   • Rolled out a software update to the edge routers running the affected version, site by site
   • Rebalanced traffic on the congested routes between AMS3 and FRA1
   • Rolled out the fix to the redundant networking equipment so a failover would not hit the bug

   Action items

   • Upgrade all global edge routers to remove the bug
   • Bring the network upgrades already done in Asia to Europe and North America

   View case
2. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed

   Linode2023

   Symptoms

   On 12 February 2023, customers in Linode's US-West region could not reach their Linodes. Santa Clara customers lost Internet connectivity after a change made at 8:45 UTC until it was reverted at 10:13 UTC, and later in the day the connection between the Fremont and Santa Clara data centers was cut completely, leaving both with degraded service or a full outage until 16:25 UTC; customers could also have trouble provisioning, powering down or rebooting services.

   Network unreachable

   Trigger

   A single link of the A-side fibre circuit between Fremont and Santa Clara went down on 11 February, the whole B-side circuit followed on 12 February, and during troubleshooting with the hardware vendor a non-intrusive command triggered a rare bug that took down all A-side links.

   Root cause

   Software defect

   Component

   Backbone network (WAN)NIC, cable and switch port link

   Contributing factor

   Overlapping failuresHarmful manual action

   Detection

   The response team began investigating when the A-side link went down on 11 February. The status page opened at 10:22 UTC on 12 February to investigate reports of connectivity issues in the US-West region.

   Remediation

   • Reverted the change that removed internet traffic from the Santa Clara border routers
   • Performed an emergency code upgrade of both circuits, restoring connectivity between the sites
   • Restored the B-side path with the hardware vendor and fiber provider, regaining full redundancy

   View case

By root cause1