Incident Library
Applied filters ComponentBGP routingSymptomNetwork unreachableClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases16 cases

6 to 10 of 16

  1. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
    Symptoms

    At about 06:00 UTC on 10 January 2024, SourceHut's monitoring raised the alarm as a layer 3 DDoS attack hit its infrastructure in the PHL datacenter, and at about 09:00 UTC SourceHut lost all access to its PHL network. PHL was the primary datacenter, and the network became unreachable for SourceHut staff and the general public alike.

    Network unreachableService outage
    Trigger

    A layer 3 DDoS attack against SourceHut's PHL infrastructure that began at about 06:00 UTC on 10 January 2024 and broadened to more of the PHL subnet on 11 January.

    Root cause
    Dependency failure
    Component
    ISP and transit networksBGP routing
    Contributing factor
    Single point of failureLimited debug accessUntested backup
    Detection

    SourceHut's monitoring systems noticed that something was wrong and raised the alarm from about 06:00 UTC on 10 January.

    Remediation
    • Got Cogent, through the colocation provider, to narrow the null route from the /24 to the targeted /32, which mostly restored service on 10 January
    • Moved DNS to an external DNS provider
    • Brought up a new primary database in AMS
    • Restored backups from FRE to AMS
    • Applied the difference between production data and the last backups by hand
    • Provisioned a new mail system and configured the mail services for it
    • Set up an OVH box that NATs traffic through OVH's DDoS-resistant network to the production subnet in AMS
    • Used a temporary subnet from the PHL network provider for out-of-band access to the PHL servers
    • Served pages.sr.ht through an out-of-band link to PHL while its object storage was migrated
    • Emailed pages.sr.ht users whose apex records pointed at the PHL subnet with instructions
    • Ran build workers on dedicated bare-metal OVH servers
    Action items
    • Provision a new standby database
    • Configure backups from AMS to FRE
    • Install a new monitoring system
    • Retire the PHL datacenter installation and ship its servers to AMS
    • Move build service back onto SourceHut-operated servers once the build servers are shipped from PHL to AMS
    • Research and implement a way to run SourceHut in an on-premise Kubernetes cluster
    • Plan a second production-ready datacenter to load balance against, able to take over full service in a datacenter outage
    Lessons

    The operational plans SourceHut kept for failures up to the loss of a whole datacenter proved important in this incident. The incident exposed a known shortcoming in redundancy and resilience that the team was working on but had not finished, and the response of the upstream network to the attack did more damage than anything else.

    View case
  2. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
    Symptoms

    From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

    Network unreachableIntermittent timeoutsHigh latency
    Trigger

    The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

    Root cause
    Software defect
    Component
    BGP routing
    Contributing factor
    Excessive blast radiusHarmful manual action
    Remediation
    • Rolled out a software update to the edge routers running the affected version, site by site
    • Rebalanced traffic on the congested routes between AMS3 and FRA1
    • Rolled out the fix to the redundant networking equipment so a failover would not hit the bug
    Action items
    • Upgrade all global edge routers to remove the bug
    • Bring the network upgrades already done in Asia to Europe and North America
    View case
  3. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023
    Symptoms

    From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.

    High latencyIntermittent timeoutsIntermittent connection failuresPacket lossNetwork unreachable
    Trigger

    A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.

    Root cause
    Configuration change
    Component
    Backbone network (WAN)Interior routing (IGP)BGP routing
    Contributing factor
    No configuration testsUnverified assumptionExcessive blast radiusHarmful manual action
    Detection

    Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.

    Remediation
    • Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss
    Action items
    • Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
    • Give on-call engineers real-time visibility of approved and unqualified activity on network devices
    • Run regular mandatory operational training and attestation of following all SOPs
    • Have the Change Advisory Board review all SOPs still pending qualification within 30 days
    View case
  4. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
    Wikimedia Foundation2023
    Symptoms

    On 10 January 2023 at 16:37 UTC Icinga reported "BGP status on cr3-eqsin is CRITICAL: BGP CRITICAL - AS1299/IPv6: Active - Telia, AS1299/IPv4: Active - Telia", and from 16:45 the hosts in Wikimedia's eqsin site went DOWN with "PING CRITICAL - Packet loss = 100%". Users normally directed to eqsin, mostly in the APAC region, could only read Wikipedia pages already cached in eqsin.

    Network unreachable
    Trigger

    Planned maintenance by the transport provider took down eqsin's second transport link while the first was still down from a long fiber cut.

    Root cause
    Dependency failure
    Component
    Backbone network (WAN)BGP routingCDN and edge network
    Contributing factor
    Single point of failureIgnored early signal
    Detection

    Automated monitoring: Icinga BGP status alerts for cr3-eqsin at 16:37 and cr2-eqsin at 16:44, followed by a flood of host-down alerts and three pages at the same time.

    Remediation
    • Depooled eqsin from DNS
    • Repooled eqsin
    Action items
    • Create a backup GRE tunnel for eqsin (T327265)
    • Ensure the long-down transport link comes back up properly once it is fixed (T322529)
    • Automatically parse maintenance notifications and alert on conflicting maintenance (T230835)
    Lessons

    The site was depooled quickly, and "depool first, investigate later" was the right attitude. A flood of host-down alerts usually means a network problem. eqsin stayed too long on a single working transport link, SREs did not catch that the downtime of the two transport links overlapped, and the provider's planned maintenance was not recorded in a task or reminder.

    View case
  5. IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs
    Wikimedia Foundation2021
    Symptoms

    On 23 November 2021 at 09:39, two minutes after a change on the eqiad core routers, alerts fired because Icinga in eqiad could not reach the public text-lb address in eqsin (Singapore). For about 12 minutes eqiad could not reach hosts in other data centers, such as the cache PoPs, through their public IP addresses.

    Network unreachable
    Trigger

    At about 09:37 a change on cr1-eqiad and cr2-eqiad, made as part of task T295672 to influence BGP route selection, removed a BGP setting that sets the BGP "MED" attribute to the OSPF cost of reaching the BGP route's next hop.

    Root cause
    Configuration change
    Component
    BGP routing
    Contributing factor
    Ineffective rollback
    Detection

    Icinga health checks against edge caching clusters alerted at 09:39; the first alert was Icinga in eqiad failing to reach the public text-lb address in eqsin.

    Remediation
    • Reverted the configuration change on both eqiad routers
    • Forced a reset of the BGP session between the two eqiad routers
    Action items
    • Updated T295672 with further detail on the incident and a proposal to adjust the BGP sessions to address the "second IXP port" issue and prevent this from happening again
    View case
Check your cluster for this

These 16 cases involve BGP routing and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Configuration change5 cases

Component
BGP routing5Interior routing (IGP)3Backbone network (WAN)2
Contributing factor
Excessive blast radius3Ineffective rollback3No configuration tests2Missed in review2Unverified assumption1Change during peak hours1Limited debug access1Harmful manual action1Overlapping failures1
Organisation
KT1Microsoft Azure1OVHcloud1PSBA1Wikimedia Foundation1
  1. IL-0400 Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default route
    PSBA2026
  2. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023
  3. IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs
    Wikimedia Foundation2021
  4. IL-0539 KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in Busan
    KT2021
  5. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF
    OVHcloud2021

Capacity exhaustion1 case

Component
BGP routingDatabaseInfiniBand fabricModel hubNetwork control plane (SDN)NIC, cable and switch port link
Contributing factor
Automation amplificationExcessive blast radius
Organisation
IONOS Cloud
  1. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026

Faulty deployment1 case

Component
BGP routingNetwork control plane (SDN)
Contributing factor
Partial rollout
Organisation
Linode
  1. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables
    Linode2025

Software defect4 cases

Component
BGP routing4Network control plane (SDN)3Backbone network (WAN)1Database1Managed Kubernetes control plane and provider API1Network switch1
Contributing factor
Excessive blast radius3Ineffective rollback2Automation amplification1Circular dependency1Limited debug access1Harmful manual action1Monitoring blind spot1Gap in automated tests1
Organisation
DigitalOcean1Google Cloud1Nebius1Scaleway1
  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
  3. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
  4. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019

Dependency failure4 cases

Component
BGP routing4Backbone network (WAN)2CDN and edge network1DDoS protection service1ISP and transit networks1Network control plane (SDN)1
Contributing factor
Limited debug access2Single point of failure2Untested backup1Ignored early signal1
Organisation
Fastmail1Packet1SourceHut1Wikimedia Foundation1
  1. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
  2. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
    Wikimedia Foundation2023
  3. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
    Packet2019
  4. IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down
    Fastmail2016

Operator error1 case

Component
Backbone network (WAN)BGP routingAuthoritative DNS servers and records
Contributing factor
Automation amplificationCircular dependencyLimited debug access
Organisation
Meta
  1. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes
    Meta2021