Cases1 case

1 to 1 of 1

1. IL-0389 DNS queries from EC2 instances in the Seoul region failed for 84 minutes as the EC2 DNS resolver fleet lost hosts

   Amazon Web Services2018

   Symptoms

   On 22 November 2018 between 8:19 AM and 9:43 AM KST, DNS queries from within EC2 instances in the Seoul region (AP-NORTHEAST-2) failed, because the EC2 DNS resolver fleet that provides recursive DNS to EC2 instances had too few healthy hosts in service. Network connectivity of EC2 instances and DNS resolution outside EC2 instances were not affected.

   DNS resolution failures

   Trigger

   A configuration update incorrectly removed the setting that specifies the minimum number of healthy hosts for the EC2 DNS resolver fleet in AP-NORTHEAST-2.

   Root cause

   Configuration change

   Component

   VPC DNS resolver

   Contributing factor

   No configuration testsNo safeguard on destructive change

   Detection

   The engineering team was alerted to the DNS resolution issue at 8:21 AM KST.

   Remediation

   • Stopped more healthy hosts from being removed from the EC2 DNS resolver fleet
   • Restored the resolver fleet's capacity to its earlier level

   Action items

   • Check that every AWS region has the correct capacity settings for the EC2 DNS resolver service
   • Add semantic configuration validation to every EC2 DNS resolver configuration update
   • Throttle how much healthy host capacity can be removed from service each hour

   View case

By root cause1