Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases21 cases

4 to 8 of 21

  1. IL-0564 Linode's Miami, Milan and Sao Paulo data centers had packet loss as traffic moved onto congested alternate paths
    Linode2025
    Symptoms

    Customers using Linode's US-MIA (Miami) and IT-MIL (Milan) data centers saw packet loss and brief performance problems on 14 September 2025 from 17:42 to 18:43 UTC. BR-GRU (Sao Paulo) had the same packet loss on 15 September 2025 from 01:44 to 04:22 UTC.

    Packet loss
    Trigger

    A gradual increase in prefixes pushed the routing between the core router and the edge router past the maximum number of permitted prefixes.

    Root cause
    Capacity exhaustion
    Component
    BGP routingBackbone network (WAN)
    Contributing factor
    –
    Remediation
    • Raised the prefix limit on all the data center routers
    Action items
    • Raise the thresholds of the sessions between Akamai's backbone and the data center ECOR routers
    • Improve the existing alerting and check its severity before limits are reached, with escalation to the right team for immediate action
    View case
  2. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025
    Symptoms

    From 16:27 UTC on 21 August 2025, users connecting to Cloudflare from servers in AWS us-east-1, or receiving connections from Cloudflare there, saw high latency, packet loss and failures to reach origins. Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC.

    Packet lossHigh latencyIntermittent timeouts
    Trigger

    At about 16:27 UTC a single customer began sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare's cache, doubling the total traffic from Cloudflare to AWS.

    Root cause
    Capacity exhaustion
    Component
    Transit gateway or network interconnectBGP routingCDN and edge network
    Contributing factor
    No headroomHarmful manual actionExcessive blast radius
    Detection

    Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC, and an incident was raised for customer impact at 17:45 UTC.

    Remediation
    • Rate limited the single customer whose traffic caused the surge
    • Took further traffic engineering actions that fully resolved the congestion
    • Asked AWS to revert its BGP prefix withdrawals, which AWS finished at 20:07 UTC
    Action items
    • Build a mechanism to deprioritise a customer's traffic when it congests the network enough to affect others
    • Speed up the Data Center Interconnect upgrades to add network capacity
    • Work with AWS so the two companies' BGP traffic engineering actions do not conflict
    • Build a traffic management system that gives each customer a network resource budget and automates the manual actions used here
    Lessons

    Cloudflare concludes that it must isolate customers better so that no single customer can take over shared network resources and degrade the platform for others, and that its network needs enough capacity to meet demand. It describes the outage as the result of insufficient network congestion management during an unusual high-traffic event.

    View case
  3. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables
    Linode2025
    Symptoms

    At about 16:38 UTC on 17 March 2025, internal alerts fired across multiple Linode data centers because various servers had been unreachable for an extended period. At the same time customers began reporting problems with several products.

    Network unreachableService outage
    Trigger

    A release of Akamai Compute's internal backend API component was rolling out, and during the rollout the API gave inconsistent responses to the route servers.

    Root cause
    Faulty deployment
    Component
    BGP routingNetwork control plane (SDN)
    Contributing factor
    Partial rollout
    Detection

    Internal alerts fired across multiple data centers showing servers unreachable for an extended period, and customers reported problems with multiple products at the same time.

    Remediation
    • Restarted the route servers in all data centers in phases, starting with the most affected regions
    Action items
    • Hold backend API releases until there is a better way to make sure route data for route servers is contiguous
    View case
  4. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
    Symptoms

    On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

    Network unreachable
    Trigger

    A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routing
    Contributing factor
    Ineffective rollbackGap in automated testsMonitoring blind spot
    Remediation
    • Restarted the host VPC component on the affected nodes one node at a time
    • Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem
    Action items
    • Fix the bug in the VPC host component itself
    • Add an alert on a diverging number of routes
    • Add an alert on the number of drops in the data plane
    • Develop a way to reset endpoints without restarting the host VPC component
    • Move the VPC's persistent e2e test virtual machines to canary hosts
    • Add persistent e2e tests for Managed Services for Kubernetes
    • Run e2e tests more often
    • Add a deployment stage in which non-critical internal services receive updates first
    Lessons

    The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

    View case
  5. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
    Symptoms

    On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

    Network unreachablePacket lossNode NotReady
    Trigger

    At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API
    Contributing factor
    Excessive blast radiusAutomation amplification
    Detection

    Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

    Remediation
    • Fixed both VPC route reflectors by manual action
    Action items
    • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
    • Run different software versions on the route reflectors so a single bug cannot stop several instances
    • Investigate the software issue that left the route reflectors in a stale state
    View case
Check your cluster for this

These 21 cases involve BGP routing and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Configuration change6 cases

Component
BGP routing6Backbone network (WAN)3Interior routing (IGP)3
Contributing factor
Excessive blast radius3No configuration tests3Ineffective rollback3Missed in review2Unverified assumption1Change during peak hours1Limited debug access1Harmful manual action1Overlapping failures1
Organisation
Cloudflare1KT1Microsoft Azure1OVHcloud1PSBA1Wikimedia Foundation1
  1. IL-0400 Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default route
    PSBA2026
  2. IL-0394 A Cloudflare router in Miami leaked IPv6 BGP routes to peers and providers for 25 minutes, congesting the backbone
    Cloudflare2026
  3. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023
  4. IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs
    Wikimedia Foundation2021
  5. IL-0539 KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in Busan
    KT2021
  6. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF
    OVHcloud2021

Capacity exhaustion3 cases

Component
BGP routing3Backbone network (WAN)1CDN and edge network1Database1InfiniBand fabric1Transit gateway or network interconnect1Model hub1Network control plane (SDN)1NIC, cable and switch port link1
Contributing factor
Excessive blast radius2Automation amplification1Harmful manual action1No headroom1
Organisation
Cloudflare1IONOS Cloud1Linode1
  1. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
  2. IL-0564 Linode's Miami, Milan and Sao Paulo data centers had packet loss as traffic moved onto congested alternate paths
    Linode2025
  3. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025

Faulty deployment1 case

Component
BGP routingNetwork control plane (SDN)
Contributing factor
Partial rollout
Organisation
Linode
  1. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables
    Linode2025

Software defect5 cases

Component
BGP routing5Network control plane (SDN)3Backbone network (WAN)1Database1Authoritative DNS servers and records1Managed Kubernetes control plane and provider API1Network switch1
Contributing factor
Excessive blast radius3Ineffective rollback2Signal buried in noise1Automation amplification1Circular dependency1Limited debug access1Harmful manual action1Monitoring blind spot1Overlapping failures1Single point of failure1Gap in automated tests1
Organisation
DigitalOcean1Gandi1Google Cloud1Nebius1Scaleway1
  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
  3. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
  4. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses
    Gandi2020
  5. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019

Dependency failure5 cases

Component
BGP routing5Backbone network (WAN)3In-memory cache1CDN and edge network1Database replication1DDoS protection service1ISP and transit networks1Network control plane (SDN)1
Contributing factor
Single point of failure3Limited debug access2Signal buried in noise1Untested backup1Ignored early signal1Retry storm1
Organisation
37signals1Fastmail1Packet1SourceHut1Wikimedia Foundation1
  1. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
  2. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
    Wikimedia Foundation2023
  3. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat
    37signals2020
  4. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
    Packet2019
  5. IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down
    Fastmail2016

Operator error1 case

Component
Backbone network (WAN)BGP routingAuthoritative DNS servers and records
Contributing factor
Automation amplificationCircular dependencyLimited debug access
Organisation
Meta
  1. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes
    Meta2021