Cases7 cases

4 to 7 of 7

1. IL-0223 A connection surge congested the devices between AWS's internal and main networks in us-east-1, breaking EC2 APIs

   Amazon Web Services2021

   Symptoms

   From 7:30 AM PST on 7 December 2021, AWS services in us-east-1 that communicate between AWS's internal network and its main network saw rising latency and errors. The AWS operations teams lost real-time monitoring data at once, and the first thing found in the logs was elevated internal DNS errors.

   Control plane lagHigh latencyHigh error rateMonitoring outageDNS resolution failures

   Trigger

   At 7:30 AM PST an automated activity that scales the capacity of one AWS service hosted in the main AWS network triggered unexpected behaviour from a large number of clients inside the internal network.

   Root cause

   Software defect

   Component

   Transit gateway or network interconnectAutomated capacity manager

   Contributing factor

   Retry stormCircular dependency

   Detection

   The congestion immediately cut off real-time monitoring data for the AWS operations teams, who turned to logs and first identified elevated internal DNS errors.

   Remediation

   • Moved internal DNS traffic away from the congested network paths
   • Isolated the top sources of traffic onto dedicated network devices
   • Disabled some heavy network traffic services, including EventBridge event delivery
   • Brought additional networking capacity online
   • Disabled the automated scaling activities that triggered the event

   Action items

   • Keep the triggering scaling activities off until all remediations are deployed
   • Fix the latent issue that kept networking clients from backing off
   • Deploy network configuration that protects the affected networking devices in a similar congestion event
   • Keep API Gateway servers healthy when the internal network is unavailable and speed up their recycling
   • Release a new Service Health Dashboard that makes service impact easier to understand
   • Run the support system actively across multiple AWS regions

   View case
2. IL-0336 AWS Direct Connect traffic into the Tokyo Region had packet loss and intermittent connectivity for about six hours

   Amazon Web Services2021

   Symptoms

   From 7:30 AM JST on 2 September 2021, AWS Direct Connect customers saw intermittent connectivity problems and elevated packet loss on traffic going to their VPCs in the Tokyo (AP-NORTHEAST-1) Region. Internal AWS alarms on Direct Connect packet loss fired at the same time.

   Packet lossIntermittent connection failures

   Trigger

   Customer traffic consistently generated packets with a very specific set of attributes and contents, and these packets set off a latent defect in the network device operating system on one layer of the Direct Connect network in Tokyo.

   Root cause

   Software defect

   Component

   Transit gateway or network interconnect

   Contributing factor

   Change with a delayed effectUnrepresentative staging

   Detection

   Internal alarms alerted AWS engineers to elevated packet loss for Direct Connect customers connecting to the Tokyo Region at 7:30 AM JST. The automation that monitors network devices also saw a higher rate of failed devices than normal and alerted engineers.

   Remediation

   • Removed the failing network devices from service so healthy devices carried the traffic, which helped only for a time
   • Reset failed devices and brought them back into service slowly, without stopping the failures
   • Disabled the new protocol in a single Availability Zone, then across the Tokyo Region

   Action items

   • Apply the change carefully to all other AWS Regions, with a new way to detect and remediate the issue before customers are affected

   View case
3. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down

   Slack2021

   Symptoms

   During the Americas' morning of 4 January 2021, an external monitoring service paged Slack because error rates were creeping up. While the investigation was starting, Slack's own dashboarding and alerting service became unavailable, so responders had to work without their usual dashboards and alerts.

   High error rateMonitoring outagePacket lossService outageScale-up failureFile descriptor exhaustionCloud API rate limit exhausted

   Trigger

   On the first working day after the holidays, Slack's traffic went from the quietest time of the year to one of its biggest days overnight, with cold client caches pulling more data than usual, and one of Slack's AWS Transit Gateways did not scale fast enough for it.

   Root cause

   Cloud provider outage

   Component

   Transit gateway or network interconnectAutomated capacity manager

   Contributing factor

   Circular dependencyAutomation amplification

   Detection

   An external monitoring service paged Slack when error rates started creeping up during the Americas' morning. AWS engineers were separately alerted to the packet drops by AWS's internal monitoring.

   Remediation

   • Disabled automated downscaling of the web tier to keep serving capacity and allow investigation
   • Cleared up the broken, partly provisioned instances that filled the autoscaling groups
   • Repaired provision-service so new web tier instances could be provisioned and enter service
   • AWS engineers increased the Transit Gateway capacity manually across all Availability Zones

   Action items

   • AWS reviews its Transit Gateway scaling algorithms for large packet-per-second increases
   • Request a preemptive upscaling of the Transit Gateways at the end of the next holiday season
   • Run the dashboard service instances in the same VPC as their database to remove the Transit Gateway dependency
   • Load test provision-service regularly
   • Reevaluate health-checking and autoscaling configurations so they cannot overload provision-service again

   Lessons

   Monitoring has to stay as independent as possible of the infrastructure it watches, or it fails when it is needed most. Here the dashboard and alerting services failed because they ran in a different VPC from their databases and so depended on the Transit Gateways. provision-service had been load tested before, but this event exceeded the earlier scale-ups.

   View case
4. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded

   PagerDuty2013sev1

   Symptoms

   On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

   High latencyHigh error rateIntermittent connection failures

   Trigger

   At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

   Root cause

   Cloud provider outage

   Component

   Transit gateway or network interconnectCoordination and key-value store

   Contributing factor

   Single point of failureUnverified assumptionMonitoring blind spot

   Detection

   The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

   Remediation

   • Restarted most of the failed coordinator processes by hand
   • The notification dispatch system reached quorum again and processed all queued notifications

   Action items

   • Add more logging and aggregate logs into a single searchable source
   • Add a process watcher that restarts failed coordinator processes automatically
   • Build a dashboard that shows inter-host connectivity
   • Train engineering staff on Cassandra and ZooKeeper
   • Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure

   Lessons

   Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

   View case

By root cause5