Cases3 cases

1 to 3 of 3

1. IL-0389 DNS queries from EC2 instances in the Seoul region failed for 84 minutes as the EC2 DNS resolver fleet lost hosts

   Amazon Web Services2018

   Symptoms

   On 22 November 2018 between 8:19 AM and 9:43 AM KST, DNS queries from within EC2 instances in the Seoul region (AP-NORTHEAST-2) failed, because the EC2 DNS resolver fleet that provides recursive DNS to EC2 instances had too few healthy hosts in service. Network connectivity of EC2 instances and DNS resolution outside EC2 instances were not affected.

   DNS resolution failures

   Trigger

   A configuration update incorrectly removed the setting that specifies the minimum number of healthy hosts for the EC2 DNS resolver fleet in AP-NORTHEAST-2.

   Root cause

   Configuration change

   Component

   VPC DNS resolver

   Contributing factor

   No configuration testsNo safeguard on destructive change

   Detection

   The engineering team was alerted to the DNS resolution issue at 8:21 AM KST.

   Remediation

   • Stopped more healthy hosts from being removed from the EC2 DNS resolver fleet
   • Restored the resolver fleet's capacity to its earlier level

   Action items

   • Check that every AWS region has the correct capacity settings for the EC2 DNS resolver service
   • Add semantic configuration validation to every EC2 DNS resolver configuration update
   • Throttle how much healthy host capacity can be removed from service each hour

   View case
2. IL-0534 Amazon S3 in US-EAST-1 could not serve GET, LIST, PUT or DELETE requests for hours while its index subsystem restarted

   Amazon Web Services2017

   Symptoms

   From 9:37AM PST on February 28th, 2017, Amazon S3 in the Northern Virginia (US-EAST-1) Region could not serve requests. AWS services in the region that store data in S3 failed with it, including the S3 console, new Amazon EC2 instance launches, Amazon EBS volumes that needed data from an S3 snapshot, and AWS Lambda.

   Service outage

   Trigger

   While debugging an S3 billing system that was progressing more slowly than expected, an authorized S3 team member followed an established playbook to remove a small number of servers from one S3 subsystem used by the billing process, and entered one input of the command incorrectly.

   Root cause

   Operator error

   Component

   Object storageCloud block storageServerless function platform

   Contributing factor

   No safeguard on destructive changeExcessive blast radiusCircular dependency

   Remediation

   • Restarted the index subsystem, which served GET, LIST and DELETE again from 12:26PM PST
   • Restarted the placement subsystem after the index subsystem worked, restoring PUT by 1:54PM PST

   Action items

   • Change the capacity removal tool to remove capacity more slowly
   • Add safeguards that stop capacity removal when it would take a subsystem below its minimum required capacity
   • Audit other operational tools for similar safety checks
   • Improve the recovery time of key S3 subsystems and start the planned further partitioning of the index subsystem into cells now
   • Run the SHD administration console across multiple AWS regions

   View case
3. IL-0224 ELB control plane APIs failed in US-East, and load balancers it then changed broke, after ELB state data was deleted

   Amazon Web Services2012

   Symptoms

   From 12:24 PM PST on 24 December 2012, the Amazon Elastic Load Balancing (ELB) control plane in the US-East Region showed high latency and error rates for API calls that manage load balancers. Running load balancers kept handling requests normally at first.

   Control plane lagHigh error rateHigh latency

   Trigger

   One of the very few developers with access to the production ELB environment inadvertently ran a maintenance process against the production ELB state data, and it logically deleted part of that data.

   Root cause

   Operator error

   Component

   Cloud load balancer

   Contributing factor

   No safeguard on destructive changeUntested backupAutomation amplification

   Detection

   The ELB team noticed the control plane API errors first; the missing state data was found only after the team investigated load balancers that had become degraded.

   Remediation

   • Disabled several ELB control plane workflows, including scaling and descaling
   • Recovered some of the affected running load balancers by hand
   • Restored a snapshot of the ELB state data from just before the deletion, using an alternate recovery process
   • Merged the restored data with the state changes since the snapshot and verified it
   • Re-enabled the ELB workflows and APIs slowly while recovering each affected load balancer

   Action items

   • Require per-incident Change Management approval for all access to production ELB state data
   • Change the data recovery process so ELB state data can be recovered much faster
   • Reprogram the ELB control plane workflows to reconcile central service data with current load balancer state

   View case

By root cause2