Incident Library
Applied filters SymptomNetwork unreachableFactorSingle point of failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

5 to 6 of 6

  1. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance
    OpenStreetMap Foundation2022
    Symptoms

    From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

    Network unreachableService outage
    Trigger

    Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

    Root cause
    Hardware failure
    Component
    Network switchData center power and cooling
    Contributing factor
    Single point of failureLimited debug accessMonitoring blind spot
    Remediation
    • Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC
    Action items
    • Acknowledge ongoing outages faster
    • Add monitoring and alerting of switch PSU and FAN states
    • Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
    • Install out-of-band access equipment with dual power feeds in Amsterdam
    • Install redundant uplinks
    • Make site failover easier, less manual and lower risk
    • Introduce an on-call pager system for the operations SRE
    • Expand the SRE team
    • Continue and formalise fail-over tests for power, network, upstream link and service outages
    View case
  2. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung
    Netlify2016
    Symptoms

    At 8am PST on Monday, August 22, 2016, Netlify got alerts that its central load balancer, the single public IP that users point A records at, had stopped responding. Rackspace had null-routed the load balancer, so all traffic to that IP hung before it reached Netlify's servers.

    Network unreachable
    Trigger

    A DDoS attack against Netlify's central load balancer in Rackspace's Chicago datacenter, after which Rackspace null-routed the load balancer's public IP.

    Root cause
    Cloud provider outage
    Component
    Cloud load balancer
    Contributing factor
    Single point of failureLimited debug access
    Detection

    Alerts at 8am PST reported that the central load balancer had stopped responding, and Rackspace then confirmed it had null-routed the load balancer.

    Remediation
    • Started a new cluster of CDN machines in Google Compute behind a Google network load balancer
    • Gave clients who had contacted support the new public IP and told them to change their A records
    • Updated the documentation and announced the new IP on Twitter, Gitter and the status page
    Action items
    • Keep the new Google-based network IP running in place of the Rackspace load balancer
    • Release Netlify's own DNS hosting, integrated with the CDN, so domains do not depend on a fixed IP
    • Fix the status page publishing process so it is updated straight away during an issue
    • Build a mailer that sends targeted mails to all paying clients during an outage
    Lessons

    Netlify judged Rackspace's load balancer too vulnerable to DDoS and Rackspace's DDoS processes not up to the standard of other cloud providers, and expected Google's anycast-based Maglev load balancer to be more resilient. In the end, the only way to protect a domain from DDoS-related outages is not to expose any fixed IP address for it, which is why sites should use the www domain with a CNAME, or Netlify DNS, rather than a single A-record IP.

    View case
Check your cluster for this

These 6 cases involve ISP and transit networks and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Dependency failure4 cases

Component
ISP and transit networks3BGP routing2Backbone network (WAN)1CDN and edge network1Database replication1
Contributing factor
Single point of failure4Untested backup1Ignored early signal1Limited debug access1Monitoring blind spot1No runbook1
Organisation
Fastmail1OpenStreetMap Foundation1SourceHut1Wikimedia Foundation1
  1. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks
    OpenStreetMap Foundation2024
  2. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
  3. IL-0259 Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customers
    Fastmail2023
  4. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
    Wikimedia Foundation2023

Hardware failure1 case

Component
Data center power and coolingNetwork switch
Contributing factor
Limited debug accessMonitoring blind spotSingle point of failure
Organisation
OpenStreetMap Foundation
  1. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance
    OpenStreetMap Foundation2022

Cloud provider outage1 case

Component
Cloud load balancer
Contributing factor
Limited debug accessSingle point of failure
Organisation
Netlify
  1. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung
    Netlify2016