Incident Library
Applied filters ComponentBackbone network (WAN)Clear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases12 cases

12 to 12 of 12

  1. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
    Packet2019
    Symptoms

    At about 08:49 UTC on 2 June 2019, Packet's internal network monitoring alerted on serious reachability and performance problems in several of Packet's global datacenter locations. The first investigation found routing loops and traffic being blackholed to specific internal and Internet destinations.

    Network unreachablePacket lossHigh latency
    Trigger

    A trans-continental US backbone link in the network of Packet's upstream provider Zayo had a hardware failure and blackholed IP traffic routed over it.

    Root cause
    Dependency failure
    Component
    Backbone network (WAN)BGP routingNetwork control plane (SDN)
    Contributing factor
    –
    Detection

    Packet's internal network monitoring alerted at 08:49 UTC on reachability issues in several global datacenter locations, and engineers confirmed them at 08:52.

    Remediation
    • Migrated traffic to an alternative network and removed Zayo from the provider mix
    • Applied route filters to return service to normal
    • Optimised specific routing policies to improve latency across the backbone with the new traffic mix
    Action items
    • Expand internal connectivity monitoring and alerting of on-call operations teams
    • Expand network automation and routing so traffic can be balanced across all available backbone paths and an operator can trigger automated re-routes across all Packet datacenters in a major provider incident
    • Make Packet's provisioning systems less dependent on third party network providers and inter-datacenter connectivity
    View case
Check your cluster for this

These 12 cases involve Backbone network (WAN) and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Configuration change3 cases

Component
Backbone network (WAN)3BGP routing3Interior routing (IGP)2
Contributing factor
Excessive blast radius2No configuration tests2Unverified assumption1Harmful manual action1Ineffective rollback1
Organisation
Cloudflare1Microsoft Azure1OVHcloud1
  1. IL-0394 A Cloudflare router in Miami leaked IPv6 BGP routes to peers and providers for 25 minutes, congesting the backbone
    Cloudflare2026
  2. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023
  3. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF
    OVHcloud2021

Capacity exhaustion1 case

Component
Backbone network (WAN)BGP routing
Contributing factor
–
Organisation
Linode
  1. IL-0564 Linode's Miami, Milan and Sao Paulo data centers had packet loss as traffic moved onto congested alternate paths
    Linode2025

Operator error2 cases

Component
Backbone network (WAN)2Authoritative DNS servers and records2BGP routing1Object storage1
Contributing factor
Automation amplification2Circular dependency1Limited debug access1
Organisation
Meta1Scaleway1
  1. IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturated
    Scaleway2025
  2. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes
    Meta2021

Software defect3 cases

Component
Backbone network (WAN)3BGP routing1Network control plane (SDN)1NIC, cable and switch port link1On-premises load balancer1
Contributing factor
Overlapping failures2Excessive blast radius1Circular dependency1Limited debug access1Harmful manual action1No headroom1Ineffective rollback1
Organisation
37signals1Google Cloud1Linode1
  1. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed
    Linode2023
  2. IL-0256 Basecamp was fully down twice in two days as hardware load balancers crashed and routers lost sync after failover
    37signals2020
  3. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019

Dependency failure3 cases

Component
Backbone network (WAN)3BGP routing3In-memory cache1CDN and edge network1Database replication1Network control plane (SDN)1
Contributing factor
Single point of failure2Signal buried in noise1Ignored early signal1Retry storm1
Organisation
37signals1Packet1Wikimedia Foundation1
  1. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
    Wikimedia Foundation2023
  2. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat
    37signals2020
  3. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
    Packet2019