Cases1 case

1 to 1 of 1

1. IL-0395 Google Cloud HTTP(S) Load Balancer gave 502s on 25% of requests for 22 minutes after reverting to a stale configuration

   Google Cloud2017

   Symptoms

   On 5 April 2017 from 01:13 to 01:35 PDT, 25% of requests to the Google Cloud HTTP(S) Load Balancer failed, and clients received 502 errors for the failed requests. Some HTTP(S) Load Balancers that had been modified recently failed 100% of their requests.

   High error rateProbe failures

   Trigger

   One replica of the master servers that control the HTTP(S) Load Balancer configuration update process lost access to Google's distributed file system and could not read recent configuration files, and mastership then passed to that replica.

   Root cause

   Software defect

   Component

   Cloud load balancer

   Contributing factor

   Automation amplification

   Detection

   Google engineers were paged at 01:22 PDT, nine minutes after the errors began.

   Remediation

   • Switched the configuration update process to a different master server at 01:34 PDT
   • Paused configuration updates to the HTTP(S) Load Balancer until 05:16 while the root cause was confirmed

   Action items

   • Configure master servers never to push HTTP(S) Load Balancer configurations more than a few hours old
   • Make Google Frontend servers reject loading a configuration file more than a few hours old
   • Improve testing of new HTTP(S) Load Balancer configurations so out-of-date configurations are flagged before reaching production
   • Fix the issue that made the master server fail to read files from Google's distributed file system
   • Fix the issue that made Google Frontends fail health checks during heavy garbage collection

   View case

By root cause1