Cases2 cases

1 to 2 of 2

1. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups

   unknown2025

   Symptoms

   CI jobs failed now and then because a hostname could not be resolved. The failures came from tensorstore, appeared on several different instances and did not repeat consistently. The error read: "./tensorstore/util/result.h:420: Status not ok: status(): UNAVAILABLE: CURL error Could not resolve hostname: Could not resolve host: company-bucket.s3.amazonaws.com".

   DNS resolution failuresIntermittent timeouts

   Trigger

   An administrative pod running with host networking called sudo in a loop. Each sudo call looked up the node's own hostname, and with ndots:5 and four search domains every lookup became five queries, for both A and AAAA records.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cacheVPC DNS resolver

   Contributing factor

   Unknown limitUnreviewed defaults

   Detection

   Reports of flaky CI jobs that could not resolve an S3 bucket hostname, followed by i/o timeout errors in the CoreDNS logs.

   Remediation

   • added 'Defaults !fqdn' in /etc/sudoers.d/00-no-fqdn (mode 440) to the container image so sudo stopped looking up the hostname

   Lessons

   Traffic from CoreDNS to the VPC DNS resolver is limited to 1024 packets per second per interface, and drops show up only as UDP timeouts; the linklocal_allowance_exceeded counter in ethtool shows them. Names that are not fully qualified are multiplied by the search domains under ndots:5, and AAAA lookups double them again. sudo looks up the host's own name on every call unless 'Defaults !fqdn' is set.

   View case
2. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds

   Airbnb2019

   Symptoms

   On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.

   High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures

   Trigger

   Several services sent a high volume of external DNS queries, all of which went through kube-dns.

   Root cause

   Capacity exhaustion

   Component

   kube-dnskube-proxyClient-side DNS resolver and cache

   Contributing factor

   Unreviewed defaults

   Detection

   Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.

   Lessons

   By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.

   View case

By root cause1