Cases1 case

1 to 1 of 1

1. IL-0243 Google's global External Proxy Load Balancing returned 404 errors for most HTTP/S endpoints for 33 minutes

   Google Cloud2021

   Symptoms

   From 09:35 to 10:08 PT on 16 November 2021, most endpoints served by Google's global External Proxy Load Balancing (GCLB) returned Google 404 errors to HTTP/S requests. For about another 1 hour and 20 minutes customers could not change their load balancing configuration.

   Service outageHigh error rate

   Trigger

   On 16 November 2021 a race condition in the configuration pipeline occurred in a cluster that patch A had not yet reached, and pushed a corrupted configuration file to GCLB.

   Root cause

   Software defect

   Component

   Cloud load balancerServerless function platform

   Contributing factor

   Unpatched known bugExcessive blast radius

   Detection

   Automated alerting notified Google engineers at 09:50 PT, at the same time as customer support requests came in.

   Remediation

   • Rolled back to the last known good configuration, which ended the 404 errors
   • Suspended customer-initiated configuration changes in GCLB
   • Validated the fix and added proctoring and monitoring before resuming changes
   • Resumed customer configuration pushes at 11:28 PT

   Action items

   • Add alerting that reports similar issues much faster
   • Add stronger automated correctness checks on configurations before they are applied
   • Speed up planned architectural changes that improve how such issues are isolated and resolved

   View case

By root cause1