IL-0048 unknown · 2025-08-12 (published)kubernetesawsnetworking
Jack Lindamood describes an investigation in a very large AWS EKS cluster. Intermittent DNS failures in CI were traced to CoreDNS forwarding too many uncached queries to the VPC's Route 53 Resolver. Traffic to that resolver goes to a link-local address, and EC2 drops packets above 1024 per second on such traffic per network interface; the linklocal_allowance_exceeded counter in ethtool confirmed the drops.
Packet captures showed that most of the uncached traffic was for EC2-style hostnames, each queried five times. Pods get a resolv.conf with ndots:5 and four search domains (app-namespace.svc.cluster.local, svc.cluster.local, cluster.local, us-east-2.compute.internal), so any name with fewer than five dots is first tried with every search domain appended. The query pattern revealed the namespace and that the source pod used host networking, which led to one administrative pod that issued A and AAAA queries for its node's hostname every few seconds.
By cutting parts of the pod's main loop, the author narrowed the traffic down to a few subprocess calls that ran commands with sudo. Running the same commands without sudo produced no DNS traffic. An ltrace of sudo showed libsudo_util.so calling gethostname and sudoers.so calling getaddrinfo on the node's hostname, so sudo itself made the lookups because of its fqdn default. Adding 'Defaults !fqdn' to the container image stopped the lookups, and uncached DNS queries to the VPC resolver almost disappeared.
CI jobs failed now and then because a hostname could not be resolved. The failures came from tensorstore, appeared on several different instances and did not repeat consistently. The error read: "./tensorstore/util/result.h:420: Status not ok: status(): UNAVAILABLE: CURL error Could not resolve hostname: Could not resolve host: company-bucket.s3.amazonaws.com".
The CoreDNS logs (kubectl logs -n kube-system deployment/coredns --tail=1000 | grep -i error) showed upstream queries timing out on the way to the VPC's Route 53 Resolver at the VPC+2 address, for example "[ERROR] plugin/errors: 2 company-bucket.s3.amazonaws.com. A: read udp 10.1.2.3:44211->10.0.0.2:53: i/o timeout". Most failing lookups were for internal hostnames such as ip-10-8-1-3.us-east-2.compute.internal, and many were AAAA queries, so every name was being looked up twice.
AWS support pointed to a hard limit of 1024 packets per second for traffic to link-local services, which includes the VPC DNS resolver. Running ethtool -S eth0 | grep linklocal_allowance_exceeded on an affected node returned a non-zero count. Because DNS runs over UDP, CoreDNS never learned that packets were dropped; it only saw timeouts.
A tcpdump of port 53 on a CoreDNS node showed, after the expected traffic, a long tail of host-name-like queries, each repeated with every search domain appended: ip-10-0-1-10.us-east-2.compute.internal.app-namespace.svc.cluster.local, ...svc.cluster.local, ...cluster.local, ...us-east-2.compute.internal.us-east-2.compute.internal, and finally the name itself. These were cache misses that CoreDNS forwarded to the VPC resolver.
Root cause Capacity exhaustion
Reports of flaky CI jobs that could not resolve an S3 bucket hostname, followed by i/o timeout errors in the CoreDNS logs.
–
Traffic from CoreDNS to the VPC DNS resolver is limited to 1024 packets per second per interface, and drops show up only as UDP timeouts; the linklocal_allowance_exceeded counter in ethtool shows them. Names that are not fully qualified are multiplied by the search domains under ndots:5, and AAAA lookups double them again. sudo looks up the host's own name on every call unless 'Defaults !fqdn' is set.
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Client-side DNS resolver and cache, CoreDNS
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Client-side DNS resolver and cache, CoreDNS
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared CoreDNS
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared CoreDNS
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared CoreDNS
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared CoreDNS
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared CoreDNS
IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019same root cause · shared Client-side DNS resolver and cache