Incident Library
Applied filters ComponentVPC DNS resolverSymptomDNS resolution failuresFactorNo configuration testsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0389 DNS queries from EC2 instances in the Seoul region failed for 84 minutes as the EC2 DNS resolver fleet lost hosts
    Amazon Web Services2018
    Symptoms

    On 22 November 2018 between 8:19 AM and 9:43 AM KST, DNS queries from within EC2 instances in the Seoul region (AP-NORTHEAST-2) failed, because the EC2 DNS resolver fleet that provides recursive DNS to EC2 instances had too few healthy hosts in service. Network connectivity of EC2 instances and DNS resolution outside EC2 instances were not affected.

    DNS resolution failures
    Trigger

    A configuration update incorrectly removed the setting that specifies the minimum number of healthy hosts for the EC2 DNS resolver fleet in AP-NORTHEAST-2.

    Root cause
    Configuration change
    Component
    VPC DNS resolver
    Contributing factor
    No configuration testsNo safeguard on destructive change
    Detection

    The engineering team was alerted to the DNS resolution issue at 8:21 AM KST.

    Remediation
    • Stopped more healthy hosts from being removed from the EC2 DNS resolver fleet
    • Restored the resolver fleet's capacity to its earlier level
    Action items
    • Check that every AWS region has the correct capacity settings for the EC2 DNS resolver service
    • Add semantic configuration validation to every EC2 DNS resolver configuration update
    • Throttle how much healthy host capacity can be removed from service each hour
    View case
Check your cluster for this

This case involves VPC DNS resolver. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
VPC DNS resolver
Contributing factor
No configuration testsNo safeguard on destructive change
Organisation
Amazon Web Services
  1. IL-0389 DNS queries from EC2 instances in the Seoul region failed for 84 minutes as the EC2 DNS resolver fleet lost hosts
    Amazon Web Services2018