Cases2 cases

1 to 2 of 2

1. IL-0243 Google's global External Proxy Load Balancing returned 404 errors for most HTTP/S endpoints for 33 minutes

   Google Cloud2021

   Symptoms

   From 09:35 to 10:08 PT on 16 November 2021, most endpoints served by Google's global External Proxy Load Balancing (GCLB) returned Google 404 errors to HTTP/S requests. For about another 1 hour and 20 minutes customers could not change their load balancing configuration.

   Service outageHigh error rate

   Trigger

   On 16 November 2021 a race condition in the configuration pipeline occurred in a cluster that patch A had not yet reached, and pushed a corrupted configuration file to GCLB.

   Root cause

   Software defect

   Component

   Cloud load balancerServerless function platform

   Contributing factor

   Unpatched known bugExcessive blast radius

   Detection

   Automated alerting notified Google engineers at 09:50 PT, at the same time as customer support requests came in.

   Remediation

   • Rolled back to the last known good configuration, which ended the 404 errors
   • Suspended customer-initiated configuration changes in GCLB
   • Validated the fix and added proctoring and monitoring before resuming changes
   • Resumed customer configuration pushes at 11:28 PT

   Action items

   • Add alerting that reports similar issues much faster
   • Add stronger automated correctness checks on configurations before they are applied
   • Speed up planned architectural changes that improve how such issues are isolated and resolved

   View case
2. IL-0534 Amazon S3 in US-EAST-1 could not serve GET, LIST, PUT or DELETE requests for hours while its index subsystem restarted

   Amazon Web Services2017

   Symptoms

   From 9:37AM PST on February 28th, 2017, Amazon S3 in the Northern Virginia (US-EAST-1) Region could not serve requests. AWS services in the region that store data in S3 failed with it, including the S3 console, new Amazon EC2 instance launches, Amazon EBS volumes that needed data from an S3 snapshot, and AWS Lambda.

   Service outage

   Trigger

   While debugging an S3 billing system that was progressing more slowly than expected, an authorized S3 team member followed an established playbook to remove a small number of servers from one S3 subsystem used by the billing process, and entered one input of the command incorrectly.

   Root cause

   Operator error

   Component

   Object storageCloud block storageServerless function platform

   Contributing factor

   No safeguard on destructive changeExcessive blast radiusCircular dependency

   Remediation

   • Restarted the index subsystem, which served GET, LIST and DELETE again from 12:26PM PST
   • Restarted the placement subsystem after the index subsystem worked, restoring PUT by 1:54PM PST

   Action items

   • Change the capacity removal tool to remove capacity more slowly
   • Add safeguards that stop capacity removal when it would take a subsystem below its minimum required capacity
   • Audit other operational tools for similar safety checks
   • Improve the recovery time of key S3 subsystems and start the planned further partitioning of the index subsystem into cells now
   • Run the SHD administration console across multiple AWS regions

   View case

By root cause2