Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
    Symptoms

    From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

    Network unreachableIntermittent timeoutsHigh latency
    Trigger

    The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

    Root cause
    Software defect
    Component
    BGP routing
    Contributing factor
    Excessive blast radiusHarmful manual action
    Remediation
    • Rolled out a software update to the edge routers running the affected version, site by site
    • Rebalanced traffic on the congested routes between AMS3 and FRA1
    • Rolled out the fix to the redundant networking equipment so a failover would not hit the bug
    Action items
    • Upgrade all global edge routers to remove the bug
    • Bring the network upgrades already done in Asia to Europe and North America
    View case
Check your cluster for this

This case involves BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
BGP routing
Contributing factor
Excessive blast radiusHarmful manual action
Organisation
DigitalOcean
  1. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023