Incident Library

Applied filters Root causeSoftware defectSymptomProbe failuresOrganisationGoogle CloudClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases2 cases

1 to 2 of 2

IL-0395 Google Cloud HTTP(S) Load Balancer gave 502s on 25% of requests for 22 minutes after reverting to a stale configuration
Google Cloud2017
Symptoms
On 5 April 2017 from 01:13 to 01:35 PDT, 25% of requests to the Google Cloud HTTP(S) Load Balancer failed, and clients received 502 errors for the failed requests. Some HTTP(S) Load Balancers that had been modified recently failed 100% of their requests.
High error rate Probe failures
Trigger
One replica of the master servers that control the HTTP(S) Load Balancer configuration update process lost access to Google's distributed file system and could not read recent configuration files, and mastership then passed to that replica.
Root cause
Software defect
Component
Cloud load balancer
Contributing factor
Automation amplification
Detection
Google engineers were paged at 01:22 PDT, nine minutes after the errors began.
Remediation
Switched the configuration update process to a different master server at 01:34 PDT
Paused configuration updates to the HTTP(S) Load Balancer until 05:16 while the root cause was confirmed
Action items
Configure master servers never to push HTTP(S) Load Balancer configurations more than a few hours old
Make Google Frontend servers reject loading a configuration file more than a few hours old
Improve testing of new HTTP(S) Load Balancer configurations so out-of-date configurations are flagged before reaching production
Fix the issue that made the master server fail to read files from Google's distributed file system
Fix the issue that made Google Frontends fail health checks during heavy garbage collection
View case
IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h
Google Cloud2017
Symptoms
On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.
Network unreachable Probe failures
Trigger
A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.
Root cause
Software defect
Component
Cloud load balancer Network control plane (SDN)
Contributing factor
Excessive blast radius Gap in automated tests
Remediation
Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
Applied the fix zone by zone between 11:36 and 12:42
Action items
Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
Improve the inefficient code path
Write new tests that check behaviour on a wider range of configurations
Accelerate replacing global propagation of address configuration with decentralized routing
Add metrics and alerting that identify this kind of issue sooner
View case

Check your cluster for this

These 2 cases involve Cloud load balancer and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component: Cloud load balancer2 Network control plane (SDN)1
Contributing factor: Automation amplification1 Excessive blast radius1 Gap in automated tests1
Organisation: Google Cloud2