Incident Library
Applied filters ComponentNetwork control plane (SDN)Clear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases12 cases

12 to 12 of 12

  1. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h
    Google Cloud2017
    Symptoms

    On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.

    Network unreachableProbe failures
    Trigger

    A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.

    Root cause
    Software defect
    Component
    Cloud load balancerNetwork control plane (SDN)
    Contributing factor
    Excessive blast radiusGap in automated tests
    Remediation
    • Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
    • Applied the fix zone by zone between 11:36 and 12:42
    Action items
    • Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
    • Improve the inefficient code path
    • Write new tests that check behaviour on a wider range of configurations
    • Accelerate replacing global propagation of address configuration with decentralized routing
    • Add metrics and alerting that identify this kind of issue sooner
    View case
Check your cluster for this

These 12 cases involve Network control plane (SDN) and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Faulty deployment4 cases

Component
Network control plane (SDN)4BGP routing1CI system1Cloud load balancer1Database1Host routes and routing rules1
Contributing factor
Excessive blast radius2Unverified assumption1No alert on limit1No staged rollout1Partial rollout1Gap in automated tests1Unsafe failure behaviour1
Organisation
Fly.io1Google Cloud1Linode1Railway1
  1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed
    Railway2026
  2. IL-0354 Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routes
    Fly.io2026
  3. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables
    Linode2025
  4. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs
    Google Cloud2023

Capacity exhaustion1 case

Component
BGP routingDatabaseInfiniBand fabricModel hubNetwork control plane (SDN)NIC, cable and switch port link
Contributing factor
Automation amplificationExcessive blast radius
Organisation
IONOS Cloud
  1. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026

Software defect5 cases

Component
Network control plane (SDN)5BGP routing3Cloud load balancer2Backbone network (WAN)1Database1Managed Kubernetes control plane and provider API1NAT gateway1Network switch1
Contributing factor
Excessive blast radius4Ineffective rollback2Gap in automated tests2Automation amplification1Circular dependency1Change with a delayed effect1Ignored early signal1Limited debug access1Monitoring blind spot1No safeguard on destructive change1
Organisation
Google Cloud3Nebius1Scaleway1
  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
  3. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped
    Google Cloud2024
  4. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019
  5. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h
    Google Cloud2017

Configuration change1 case

Component
CDN and edge networkData center power and coolingDDoS protection serviceNetwork control plane (SDN)
Contributing factor
Excessive blast radiusMonitoring blind spotOverlapping failures
Organisation
Microsoft Azure
  1. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours
    Microsoft Azure2024

Dependency failure1 case

Component
Backbone network (WAN)BGP routingNetwork control plane (SDN)
Contributing factor
–
Organisation
Packet
  1. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
    Packet2019