IL-0389 Amazon Web Services · 2018-11-22awsnetworking

DNS queries from EC2 instances in the Seoul region failed for 84 minutes as the EC2 DNS resolver fleet lost hosts

A configuration update removed the minimum healthy hosts setting of the EC2 DNS resolver fleet in the Seoul region by mistake. Without it, the setting was read as a very low default value, the fleet kept fewer healthy hosts in service, and DNS queries from EC2 instances began to fail at 8:19 AM KST. Engineers were alerted at 8:21 AM and identified the root cause at 8:48 AM. The first step, which took another 15 minutes, was to stop more healthy hosts from being removed from service. Restoring capacity to its earlier level took most of the recovery time, and DNS queries from EC2 instances fully recovered at 9:43 AM KST.

Severity: –
Scope: DNS resolution from within EC2 instances; EC2 network connectivity and DNS resolution outside EC2 instances were not affected
Users affected: –
Region: AP-NORTHEAST-2 (Seoul)
Time to detect: 2m
Time to resolve: 1h 24m

Symptoms

On 22 November 2018 between 8:19 AM and 9:43 AM KST, DNS queries from within EC2 instances in the Seoul region (AP-NORTHEAST-2) failed, because the EC2 DNS resolver fleet that provides recursive DNS to EC2 instances had too few healthy hosts in service. Network connectivity of EC2 instances and DNS resolution outside EC2 instances were not affected.

Symptom DNS resolution failures

Root cause

Root cause Configuration change

Component VPC DNS resolver

Contributing factor No configuration tests No safeguard on destructive change

Trigger: A configuration update incorrectly removed the setting that specifies the minimum number of healthy hosts for the EC2 DNS resolver fleet in AP-NORTHEAST-2.

Detection

The engineering team was alerted to the DNS resolution issue at 8:21 AM KST.

Remediation

Stopped more healthy hosts from being removed from the EC2 DNS resolver fleet
Restored the resolver fleet's capacity to its earlier level

Action items

Check that every AWS region has the correct capacity settings for the EC2 DNS resolver service
Add semantic configuration validation to every EC2 DNS resolver configuration update
Throttle how much healthy host capacity can be removed from service each hour

Lessons

–

Sources

https://aws.amazon.com/message/74876-2/ · official-status · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, VPC DNS resolver among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0389
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0389/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.