Cases2 cases

1 to 2 of 2

1. IL-0535 An internal Kinesis Data Streams cell in US-EAST-1 degraded, raising errors in CloudWatch Logs, S3 events and Lambda

   Amazon Web Services2024

   Symptoms

   From 2:45 PM PDT on July 30th, 2024, several AWS services in the Northern Virginia (US-EAST-1) Region had higher latency and error rates: CloudWatch Logs, Amazon Data Firehose, the event framework that publishes Amazon S3 events, Amazon ECS, AWS Lambda, Amazon Redshift and AWS Glue. All of them used one internal cell of Amazon Kinesis Data Streams, whose requests had started to see elevated latency and errors.

   High latencyHigh error rateMonitoring outageProbe failures

   Trigger

   At 9:09 AM PDT on July 30th a routine deployment started in one Availability Zone of the Kinesis cell, and taking hosts in and out of service during the deployment set off the failure.

   Root cause

   Software defect

   Component

   Message brokerServerless function platformObject storage

   Contributing factor

   Unrepresentative stagingAutomation amplification

   Remediation

   • Deployed a change that shed load by reducing incoming stream volume from less time-sensitive internal workloads
   • Added capacity to the subsystem that provisions secure connections for Kinesis requests
   • Added more capacity at 6:32 PM PDT to further improve connection provisioning

   Action items

   • Keep the added connection provisioning capacity, load-shedding tooling and changed connection limits in place in US-EAST-1
   • Apply the same changes in every other region that has cells with the new Kinesis Data Streams architecture
   • Update the cell management system to better manage the workload that triggered the issue

   View case
2. IL-0223 A connection surge congested the devices between AWS's internal and main networks in us-east-1, breaking EC2 APIs

   Amazon Web Services2021

   Symptoms

   From 7:30 AM PST on 7 December 2021, AWS services in us-east-1 that communicate between AWS's internal network and its main network saw rising latency and errors. The AWS operations teams lost real-time monitoring data at once, and the first thing found in the logs was elevated internal DNS errors.

   Control plane lagHigh latencyHigh error rateMonitoring outageDNS resolution failures

   Trigger

   At 7:30 AM PST an automated activity that scales the capacity of one AWS service hosted in the main AWS network triggered unexpected behaviour from a large number of clients inside the internal network.

   Root cause

   Software defect

   Component

   Transit gateway or network interconnectAutomated capacity manager

   Contributing factor

   Retry stormCircular dependency

   Detection

   The congestion immediately cut off real-time monitoring data for the AWS operations teams, who turned to logs and first identified elevated internal DNS errors.

   Remediation

   • Moved internal DNS traffic away from the congested network paths
   • Isolated the top sources of traffic onto dedicated network devices
   • Disabled some heavy network traffic services, including EventBridge event delivery
   • Brought additional networking capacity online
   • Disabled the automated scaling activities that triggered the event

   Action items

   • Keep the triggering scaling activities off until all remediations are deployed
   • Fix the latent issue that kept networking clients from backing off
   • Deploy network configuration that protects the affected networking devices in a similar congestion event
   • Keep API Gateway servers healthy when the internal network is unavailable and speed up their recycling
   • Release a new Service Health Dashboard that makes service impact easier to understand
   • Run the support system actively across multiple AWS regions

   View case

By root cause1