Incident Library
Applied filters Root causeSoftware defectFactorGap in automated testsOrganisationGoogle CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests
    Google Cloud2018
    Symptoms

    From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.

    High error rateConnection resetsHigh latency
    Trigger

    A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.

    Root cause
    Software defect
    Component
    Cloud load balancerCDN and edge network
    Contributing factor
    Gap in automated testsExcessive blast radius
    Detection

    Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.

    Remediation
    • Reverted the configuration change at 12:44 PDT, which stopped the GFE restarts
    Action items
    • Add safeguards that disable features not yet in service
    • Harden the GFE testing stack against latent bugs in production binaries that make a task restart
    • Isolate the shards of GFE pools further from each other to reduce the scope of failures
    • Build a consolidated dashboard of all configuration changes for GFE pools
    View case
  2. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h
    Google Cloud2017
    Symptoms

    On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.

    Network unreachableProbe failures
    Trigger

    A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.

    Root cause
    Software defect
    Component
    Cloud load balancerNetwork control plane (SDN)
    Contributing factor
    Excessive blast radiusGap in automated tests
    Remediation
    • Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
    • Applied the fix zone by zone between 11:36 and 12:42
    Action items
    • Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
    • Improve the inefficient code path
    • Write new tests that check behaviour on a wider range of configurations
    • Accelerate replacing global propagation of address configuration with decentralized routing
    • Add metrics and alerting that identify this kind of issue sooner
    View case
Check your cluster for this

These 2 cases involve Cloud load balancer and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Cloud load balancer2CDN and edge network1Network control plane (SDN)1
Contributing factor
Excessive blast radius2Gap in automated tests2
Organisation
Google Cloud2
  1. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests
    Google Cloud2018
  2. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h
    Google Cloud2017