Cases8 cases

7 to 8 of 8

1. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests

   Google Cloud2018

   Symptoms

   From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.

   High error rateConnection resetsHigh latency

   Trigger

   A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.

   Root cause

   Software defect

   Component

   Cloud load balancerCDN and edge network

   Contributing factor

   Gap in automated testsExcessive blast radius

   Detection

   Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.

   Remediation

   • Reverted the configuration change at 12:44 PDT, which stopped the GFE restarts

   Action items

   • Add safeguards that disable features not yet in service
   • Harden the GFE testing stack against latent bugs in production binaries that make a task restart
   • Isolate the shards of GFE pools further from each other to reduce the scope of failures
   • Build a consolidated dashboard of all configuration changes for GFE pools

   View case
2. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h

   Google Cloud2017

   Symptoms

   On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.

   Network unreachableProbe failures

   Trigger

   A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.

   Root cause

   Software defect

   Component

   Cloud load balancerNetwork control plane (SDN)

   Contributing factor

   Excessive blast radiusGap in automated tests

   Remediation

   • Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
   • Applied the fix zone by zone between 11:36 and 12:42

   Action items

   • Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
   • Improve the inefficient code path
   • Write new tests that check behaviour on a wider range of configurations
   • Accelerate replacing global propagation of address configuration with decentralized routing
   • Add metrics and alerting that identify this kind of issue sooner

   View case

By root cause1