Cases1 case

1 to 1 of 1

1. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests

   Google Cloud2018

   Symptoms

   From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.

   High error rateConnection resetsHigh latency

   Trigger

   A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.

   Root cause

   Software defect

   Component

   Cloud load balancerCDN and edge network

   Contributing factor

   Gap in automated testsExcessive blast radius

   Detection

   Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.

   Remediation

   • Reverted the configuration change at 12:44 PDT, which stopped the GFE restarts

   Action items

   • Add safeguards that disable features not yet in service
   • Harden the GFE testing stack against latent bugs in production binaries that make a task restart
   • Isolate the shards of GFE pools further from each other to reduce the scope of failures
   • Build a consolidated dashboard of all configuration changes for GFE pools

   View case

By root cause1