Incident Library

IL-0342 DigitalOcean · 2023-08-09networking

DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed

Some of DigitalOcean's edge routers ran an operating system version with a bug: when the router received an RPKI-related routing update, its routing process could crash. Each of the AMS3, LON1 and SFO2 sites was connected to the Internet redundantly through several devices, but all of them ran software with the bug, so one trigger took down the entire Internet edge of a site. The routing processes restarted automatically, yet reprocessing and updating the global routing tables took the next few minutes, so each crash became an outage of about 10 to 15 minutes. This happened in three waves on 9 August 2023, the last around 12:15 UTC.

After an investigation by the router vendor, DigitalOcean rolled out a software update to the edge routers running the affected version, starting with AMS3, which was stable at 12:27 UTC, followed by SFO2 and LON1, which was stable at 13:29 UTC, and then to the redundant networking equipment so that a failover would not hit the same bug. The emergency upgrade caused additional reachability problems while traffic was shifted off circuits, adding disruptions over a total of 1 to 2 hours. SFO2 was the last site declared stable, at 14:23 UTC. DigitalOcean later upgraded all of its global edge routers.

Severity
–
Scope
Internet access for the AMS3, LON1 and SFO2 datacenters: Droplets and Droplet-based services such as Load Balancers, Kubernetes and Database clusters, and event processing on them; latency and connectivity errors on routes between AMS3 and FRA1
Users affected
–
Region
AMS3, LON1, SFO2
Time to detect
1m
Time to resolve
4h 28m

Symptoms

From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

Each datacenter had its own pattern. LON1 had total Internet outages from 09:55 to 10:11 and from 11:05 to 11:15, and a third wave from 12:13 to 12:21 that hit 90% of customer traffic. AMS3 had only partial outages, affecting up to 40% or 50% of customer traffic, between 09:50 and 12:27. SFO2 was fully down from 09:55 to 10:11, from 11:48 to 12:27 and from 12:41 to 12:51, with partial outages in between and at 14:21 to 14:23.

Event processing was affected as well: creating and deleting Droplets, powering them off and on, taking snapshots and similar actions on Droplets and on Droplet-based products were delayed or failed. Separately, congestion appeared on network routes between AMS3 and FRA1, which DigitalOcean describes as a different issue made worse by this one; it brought latency and connectivity errors to services in those regions and on paths through them, including services in LON1, until traffic was rebalanced.

The routing processes on the edge routers restarted on their own after each crash, but reprocessing the global routing tables took several minutes each time, which set the length of each wave. The emergency software upgrade that ended the crashes itself caused further short outages, as traffic was moved off circuits and the edge routers rejoined the Internet. When DigitalOcean announced that the rollout to the redundant equipment was complete, it also reported a brief period in which SFO2 was affected while traffic reconverged.

Symptom Network unreachableIntermittent timeoutsHigh latency

Root cause

Root cause Software defect

Component BGP routing
Contributing factor Excessive blast radiusHarmful manual action
Trigger
The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

Detection

–

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, BGP routing among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.