Cases2 cases

1 to 2 of 2

1. IL-0535 An internal Kinesis Data Streams cell in US-EAST-1 degraded, raising errors in CloudWatch Logs, S3 events and Lambda

   Amazon Web Services2024

   Symptoms

   From 2:45 PM PDT on July 30th, 2024, several AWS services in the Northern Virginia (US-EAST-1) Region had higher latency and error rates: CloudWatch Logs, Amazon Data Firehose, the event framework that publishes Amazon S3 events, Amazon ECS, AWS Lambda, Amazon Redshift and AWS Glue. All of them used one internal cell of Amazon Kinesis Data Streams, whose requests had started to see elevated latency and errors.

   High latencyHigh error rateMonitoring outageProbe failures

   Trigger

   At 9:09 AM PDT on July 30th a routine deployment started in one Availability Zone of the Kinesis cell, and taking hosts in and out of service during the deployment set off the failure.

   Root cause

   Software defect

   Component

   Message brokerServerless function platformObject storage

   Contributing factor

   Unrepresentative stagingAutomation amplification

   Remediation

   • Deployed a change that shed load by reducing incoming stream volume from less time-sensitive internal workloads
   • Added capacity to the subsystem that provisions secure connections for Kinesis requests
   • Added more capacity at 6:32 PM PDT to further improve connection provisioning

   Action items

   • Keep the added connection provisioning capacity, load-shedding tooling and changed connection limits in place in US-EAST-1
   • Apply the same changes in every other region that has cells with the new Kinesis Data Streams architecture
   • Update the cell management system to better manage the workload that triggered the issue

   View case
2. IL-0534 Amazon S3 in US-EAST-1 could not serve GET, LIST, PUT or DELETE requests for hours while its index subsystem restarted

   Amazon Web Services2017

   Symptoms

   From 9:37AM PST on February 28th, 2017, Amazon S3 in the Northern Virginia (US-EAST-1) Region could not serve requests. AWS services in the region that store data in S3 failed with it, including the S3 console, new Amazon EC2 instance launches, Amazon EBS volumes that needed data from an S3 snapshot, and AWS Lambda.

   Service outage

   Trigger

   While debugging an S3 billing system that was progressing more slowly than expected, an authorized S3 team member followed an established playbook to remove a small number of servers from one S3 subsystem used by the billing process, and entered one input of the command incorrectly.

   Root cause

   Operator error

   Component

   Object storageCloud block storageServerless function platform

   Contributing factor

   No safeguard on destructive changeExcessive blast radiusCircular dependency

   Remediation

   • Restarted the index subsystem, which served GET, LIST and DELETE again from 12:26PM PST
   • Restarted the placement subsystem after the index subsystem worked, restoring PUT by 1:54PM PST

   Action items

   • Change the capacity removal tool to remove capacity more slowly
   • Add safeguards that stop capacity removal when it would take a subsystem below its minimum required capacity
   • Audit other operational tools for similar safety checks
   • Improve the recovery time of key S3 subsystems and start the planned further partitioning of the index subsystem into cells now
   • Run the SHD administration console across multiple AWS regions

   View case

By root cause2