Cases3 cases

1 to 3 of 3

1. IL-0243 Google's global External Proxy Load Balancing returned 404 errors for most HTTP/S endpoints for 33 minutes

   Google Cloud2021

   Symptoms

   From 09:35 to 10:08 PT on 16 November 2021, most endpoints served by Google's global External Proxy Load Balancing (GCLB) returned Google 404 errors to HTTP/S requests. For about another 1 hour and 20 minutes customers could not change their load balancing configuration.

   Service outageHigh error rate

   Trigger

   On 16 November 2021 a race condition in the configuration pipeline occurred in a cluster that patch A had not yet reached, and pushed a corrupted configuration file to GCLB.

   Root cause

   Software defect

   Component

   Cloud load balancerServerless function platform

   Contributing factor

   Unpatched known bugExcessive blast radius

   Detection

   Automated alerting notified Google engineers at 09:50 PT, at the same time as customer support requests came in.

   Remediation

   • Rolled back to the last known good configuration, which ended the 404 errors
   • Suspended customer-initiated configuration changes in GCLB
   • Validated the fix and added proctoring and monitoring before resuming changes
   • Resumed customer configuration pushes at 11:28 PT

   Action items

   • Add alerting that reports similar issues much faster
   • Add stronger automated correctness checks on configurations before they are applied
   • Speed up planned architectural changes that improve how such issues are isolated and resolved

   View case
2. IL-0384 Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failed

   Amazon Web Services2020

   Symptoms

   At 5:15 AM PST on 25 November 2020, alarms fired for errors on putting and getting Amazon Kinesis records in the Northern Virginia (US-EAST-1) Region. From the same time CloudWatch, which processes metric and log data through Kinesis Data Streams, and CloudWatch Events and EventBridge returned more errors.

   High error rateHigh latencyMonitoring outage

   Trigger

   A relatively small addition of capacity to the Kinesis front-end fleet, made between 2:44 AM and 3:47 AM PST.

   Root cause

   Capacity exhaustion

   Component

   Message brokerServerless function platformTelemetry agent

   Contributing factor

   Unknown limitExcessive blast radiusCircular dependency

   Detection

   Alarms for errors on putting and getting Kinesis records fired at 5:15 AM PST, and the teams began reviewing logs.

   Remediation

   • Removed the newly added front-end capacity
   • Added a configuration so front-end servers load bootstrap data from the authoritative metadata store instead of neighbouring servers
   • Restarted the front-end fleet gradually, a few hundred servers per hour from 10:07 AM PST
   • Added Cognito capacity to buffer more calls to Kinesis
   • Deployed a change that reduced Cognito's dependency on Kinesis
   • Mitigated the memory contention on the Lambda invocation hosts
   • Informed customers through the backup Service Health Dashboard tool, the Personal Health Dashboard and a global banner

   Action items

   • Move the Kinesis front-end to larger CPU and memory servers so fewer servers, and fewer threads per server, are needed
   • Add fine-grained alarming for thread consumption
   • Finish testing a higher operating system thread count limit
   • Greatly improve the cold-start time of the front-end fleet
   • Move the front-end server cache to a dedicated fleet
   • Move a few large AWS services such as CloudWatch to a separate, partitioned front-end fleet
   • Accelerate the cellularization of the front-end fleet to match the back-end
   • Change the Cognito webservers so they sustain Kinesis API errors without exhausting their buffers
   • Keep 3 hours of metric data in the CloudWatch local metrics data store, first in US-EAST-1 and then globally
   • Train support engineers regularly on the backup tool for posting to the Service Health Dashboard

   Lessons

   In the Kinesis front-end, the number of threads each server keeps grows directly with the number of servers in the fleet, so adding servers can push every server past an operating system limit at the same time. AWS sees cellularization of the front-end, already done for the back-end, as the way to keep the shard-map cache within a tested and operated range and to guard against unknown scaling limits.

   View case
3. IL-0534 Amazon S3 in US-EAST-1 could not serve GET, LIST, PUT or DELETE requests for hours while its index subsystem restarted

   Amazon Web Services2017

   Symptoms

   From 9:37AM PST on February 28th, 2017, Amazon S3 in the Northern Virginia (US-EAST-1) Region could not serve requests. AWS services in the region that store data in S3 failed with it, including the S3 console, new Amazon EC2 instance launches, Amazon EBS volumes that needed data from an S3 snapshot, and AWS Lambda.

   Service outage

   Trigger

   While debugging an S3 billing system that was progressing more slowly than expected, an authorized S3 team member followed an established playbook to remove a small number of servers from one S3 subsystem used by the billing process, and entered one input of the command incorrectly.

   Root cause

   Operator error

   Component

   Object storageCloud block storageServerless function platform

   Contributing factor

   No safeguard on destructive changeExcessive blast radiusCircular dependency

   Remediation

   • Restarted the index subsystem, which served GET, LIST and DELETE again from 12:26PM PST
   • Restarted the placement subsystem after the index subsystem worked, restoring PUT by 1:54PM PST

   Action items

   • Change the capacity removal tool to remove capacity more slowly
   • Add safeguards that stop capacity removal when it would take a subsystem below its minimum required capacity
   • Audit other operational tools for similar safety checks
   • Improve the recovery time of key S3 subsystems and start the planned further partitioning of the index subsystem into cells now
   • Run the SHD administration console across multiple AWS regions

   View case

By root cause3