Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases21 cases

6 to 10 of 21

  1. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables
    Linode2025
    Symptoms

    At about 16:38 UTC on 17 March 2025, internal alerts fired across multiple Linode data centers because various servers had been unreachable for an extended period. At the same time customers began reporting problems with several products.

    Network unreachableService outage
    Trigger

    A release of Akamai Compute's internal backend API component was rolling out, and during the rollout the API gave inconsistent responses to the route servers.

    Root cause
    Faulty deployment
    Component
    BGP routingNetwork control plane (SDN)
    Contributing factor
    Partial rollout
    Detection

    Internal alerts fired across multiple data centers showing servers unreachable for an extended period, and customers reported problems with multiple products at the same time.

    Remediation
    • Restarted the route servers in all data centers in phases, starting with the most affected regions
    Action items
    • Hold backend API releases until there is a better way to make sure route data for route servers is contiguous
    View case
  2. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
    Symptoms

    On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

    Network unreachable
    Trigger

    A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routing
    Contributing factor
    Ineffective rollbackGap in automated testsMonitoring blind spot
    Remediation
    • Restarted the host VPC component on the affected nodes one node at a time
    • Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem
    Action items
    • Fix the bug in the VPC host component itself
    • Add an alert on a diverging number of routes
    • Add an alert on the number of drops in the data plane
    • Develop a way to reset endpoints without restarting the host VPC component
    • Move the VPC's persistent e2e test virtual machines to canary hosts
    • Add persistent e2e tests for Managed Services for Kubernetes
    • Run e2e tests more often
    • Add a deployment stage in which non-critical internal services receive updates first
    Lessons

    The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

    View case
  3. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
    Symptoms

    On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

    Network unreachablePacket lossNode NotReady
    Trigger

    At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API
    Contributing factor
    Excessive blast radiusAutomation amplification
    Detection

    Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

    Remediation
    • Fixed both VPC route reflectors by manual action
    Action items
    • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
    • Run different software versions on the route reflectors so a single bug cannot stop several instances
    • Investigate the software issue that left the route reflectors in a stale state
    View case
  4. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
    Symptoms

    At about 06:00 UTC on 10 January 2024, SourceHut's monitoring raised the alarm as a layer 3 DDoS attack hit its infrastructure in the PHL datacenter, and at about 09:00 UTC SourceHut lost all access to its PHL network. PHL was the primary datacenter, and the network became unreachable for SourceHut staff and the general public alike.

    Network unreachableService outage
    Trigger

    A layer 3 DDoS attack against SourceHut's PHL infrastructure that began at about 06:00 UTC on 10 January 2024 and broadened to more of the PHL subnet on 11 January.

    Root cause
    Dependency failure
    Component
    ISP and transit networksBGP routing
    Contributing factor
    Single point of failureLimited debug accessUntested backup
    Detection

    SourceHut's monitoring systems noticed that something was wrong and raised the alarm from about 06:00 UTC on 10 January.

    Remediation
    • Got Cogent, through the colocation provider, to narrow the null route from the /24 to the targeted /32, which mostly restored service on 10 January
    • Moved DNS to an external DNS provider
    • Brought up a new primary database in AMS
    • Restored backups from FRE to AMS
    • Applied the difference between production data and the last backups by hand
    • Provisioned a new mail system and configured the mail services for it
    • Set up an OVH box that NATs traffic through OVH's DDoS-resistant network to the production subnet in AMS
    • Used a temporary subnet from the PHL network provider for out-of-band access to the PHL servers
    • Served pages.sr.ht through an out-of-band link to PHL while its object storage was migrated
    • Emailed pages.sr.ht users whose apex records pointed at the PHL subnet with instructions
    • Ran build workers on dedicated bare-metal OVH servers
    Action items
    • Provision a new standby database
    • Configure backups from AMS to FRE
    • Install a new monitoring system
    • Retire the PHL datacenter installation and ship its servers to AMS
    • Move build service back onto SourceHut-operated servers once the build servers are shipped from PHL to AMS
    • Research and implement a way to run SourceHut in an on-premise Kubernetes cluster
    • Plan a second production-ready datacenter to load balance against, able to take over full service in a datacenter outage
    Lessons

    The operational plans SourceHut kept for failures up to the loss of a whole datacenter proved important in this incident. The incident exposed a known shortcoming in redundancy and resilience that the team was working on but had not finished, and the response of the upstream network to the attack did more damage than anything else.

    View case
  5. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
    Symptoms

    From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

    Network unreachableIntermittent timeoutsHigh latency
    Trigger

    The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

    Root cause
    Software defect
    Component
    BGP routing
    Contributing factor
    Excessive blast radiusHarmful manual action
    Remediation
    • Rolled out a software update to the edge routers running the affected version, site by site
    • Rebalanced traffic on the congested routes between AMS3 and FRA1
    • Rolled out the fix to the redundant networking equipment so a failover would not hit the bug
    Action items
    • Upgrade all global edge routers to remove the bug
    • Bring the network upgrades already done in Asia to Europe and North America
    View case
Check your cluster for this

These 21 cases involve BGP routing and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Configuration change6 cases

Component
BGP routing6Backbone network (WAN)3Interior routing (IGP)3
Contributing factor
Excessive blast radius3No configuration tests3Ineffective rollback3Missed in review2Unverified assumption1Change during peak hours1Limited debug access1Harmful manual action1Overlapping failures1
Organisation
Cloudflare1KT1Microsoft Azure1OVHcloud1PSBA1Wikimedia Foundation1
  1. IL-0400 Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default route
    PSBA2026
  2. IL-0394 A Cloudflare router in Miami leaked IPv6 BGP routes to peers and providers for 25 minutes, congesting the backbone
    Cloudflare2026
  3. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023
  4. IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs
    Wikimedia Foundation2021
  5. IL-0539 KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in Busan
    KT2021
  6. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF
    OVHcloud2021

Capacity exhaustion3 cases

Component
BGP routing3Backbone network (WAN)1CDN and edge network1Database1InfiniBand fabric1Transit gateway or network interconnect1Model hub1Network control plane (SDN)1NIC, cable and switch port link1
Contributing factor
Excessive blast radius2Automation amplification1Harmful manual action1No headroom1
Organisation
Cloudflare1IONOS Cloud1Linode1
  1. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
  2. IL-0564 Linode's Miami, Milan and Sao Paulo data centers had packet loss as traffic moved onto congested alternate paths
    Linode2025
  3. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025

Faulty deployment1 case

Component
BGP routingNetwork control plane (SDN)
Contributing factor
Partial rollout
Organisation
Linode
  1. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables
    Linode2025

Software defect5 cases

Component
BGP routing5Network control plane (SDN)3Backbone network (WAN)1Database1Authoritative DNS servers and records1Managed Kubernetes control plane and provider API1Network switch1
Contributing factor
Excessive blast radius3Ineffective rollback2Signal buried in noise1Automation amplification1Circular dependency1Limited debug access1Harmful manual action1Monitoring blind spot1Overlapping failures1Single point of failure1Gap in automated tests1
Organisation
DigitalOcean1Gandi1Google Cloud1Nebius1Scaleway1
  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
  3. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
  4. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses
    Gandi2020
  5. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019

Dependency failure5 cases

Component
BGP routing5Backbone network (WAN)3In-memory cache1CDN and edge network1Database replication1DDoS protection service1ISP and transit networks1Network control plane (SDN)1
Contributing factor
Single point of failure3Limited debug access2Signal buried in noise1Untested backup1Ignored early signal1Retry storm1
Organisation
37signals1Fastmail1Packet1SourceHut1Wikimedia Foundation1
  1. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
  2. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
    Wikimedia Foundation2023
  3. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat
    37signals2020
  4. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
    Packet2019
  5. IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down
    Fastmail2016

Operator error1 case

Component
Backbone network (WAN)BGP routingAuthoritative DNS servers and records
Contributing factor
Automation amplificationCircular dependencyLimited debug access
Organisation
Meta
  1. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes
    Meta2021