Incident Library
Applied filters ComponentBackbone network (WAN)SymptomNetwork unreachableFactorHarmful manual actionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed
    Linode2023
    Symptoms

    On 12 February 2023, customers in Linode's US-West region could not reach their Linodes. Santa Clara customers lost Internet connectivity after a change made at 8:45 UTC until it was reverted at 10:13 UTC, and later in the day the connection between the Fremont and Santa Clara data centers was cut completely, leaving both with degraded service or a full outage until 16:25 UTC; customers could also have trouble provisioning, powering down or rebooting services.

    Network unreachable
    Trigger

    A single link of the A-side fibre circuit between Fremont and Santa Clara went down on 11 February, the whole B-side circuit followed on 12 February, and during troubleshooting with the hardware vendor a non-intrusive command triggered a rare bug that took down all A-side links.

    Root cause
    Software defect
    Component
    Backbone network (WAN)NIC, cable and switch port link
    Contributing factor
    Overlapping failuresHarmful manual action
    Detection

    The response team began investigating when the A-side link went down on 11 February. The status page opened at 10:22 UTC on 12 February to investigate reports of connectivity issues in the US-West region.

    Remediation
    • Reverted the change that removed internet traffic from the Santa Clara border routers
    • Performed an emergency code upgrade of both circuits, restoring connectivity between the sites
    • Restored the B-side path with the hardware vendor and fiber provider, regaining full redundancy
    View case
  2. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023
    Symptoms

    From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.

    High latencyIntermittent timeoutsIntermittent connection failuresPacket lossNetwork unreachable
    Trigger

    A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.

    Root cause
    Configuration change
    Component
    Backbone network (WAN)Interior routing (IGP)BGP routing
    Contributing factor
    No configuration testsUnverified assumptionExcessive blast radiusHarmful manual action
    Detection

    Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.

    Remediation
    • Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss
    Action items
    • Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
    • Give on-call engineers real-time visibility of approved and unqualified activity on network devices
    • Run regular mandatory operational training and attestation of following all SOPs
    • Have the Change Advisory Board review all SOPs still pending qualification within 30 days
    View case
Check your cluster for this

These 2 cases involve Backbone network (WAN) and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Software defect1 case

Component
Backbone network (WAN)NIC, cable and switch port link
Contributing factor
Harmful manual actionOverlapping failures
Organisation
Linode
  1. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed
    Linode2023

Configuration change1 case

Component
Backbone network (WAN)BGP routingInterior routing (IGP)
Contributing factor
Unverified assumptionExcessive blast radiusHarmful manual actionNo configuration tests
Organisation
Microsoft Azure
  1. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023