Incident Library
Applied filters SymptomProbe failuresFactorExcessive blast radiusOrganisationGoogle CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h
    Google Cloud2017
    Symptoms

    On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.

    Network unreachableProbe failures
    Trigger

    A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.

    Root cause
    Software defect
    Component
    Cloud load balancerNetwork control plane (SDN)
    Contributing factor
    Excessive blast radiusGap in automated tests
    Remediation
    • Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
    • Applied the fix zone by zone between 11:36 and 12:42
    Action items
    • Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
    • Improve the inefficient code path
    • Write new tests that check behaviour on a wider range of configurations
    • Accelerate replacing global propagation of address configuration with decentralized routing
    • Add metrics and alerting that identify this kind of issue sooner
    View case
Check your cluster for this

This case involves Cloud load balancer and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Cloud load balancerNetwork control plane (SDN)
Contributing factor
Excessive blast radiusGap in automated tests
Organisation
Google Cloud
  1. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h
    Google Cloud2017