Cases7 cases

1 to 5 of 7

1. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups

   unknown2025

   Symptoms

   CI jobs failed now and then because a hostname could not be resolved. The failures came from tensorstore, appeared on several different instances and did not repeat consistently. The error read: "./tensorstore/util/result.h:420: Status not ok: status(): UNAVAILABLE: CURL error Could not resolve hostname: Could not resolve host: company-bucket.s3.amazonaws.com".

   DNS resolution failuresIntermittent timeouts

   Trigger

   An administrative pod running with host networking called sudo in a loop. Each sudo call looked up the node's own hostname, and with ndots:5 and four search domains every lookup became five queries, for both A and AAAA records.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cacheVPC DNS resolver

   Contributing factor

   Unknown limitUnreviewed defaults

   Detection

   Reports of flaky CI jobs that could not resolve an S3 bucket hostname, followed by i/o timeout errors in the CoreDNS logs.

   Remediation

   • added 'Defaults !fqdn' in /etc/sudoers.d/00-no-fqdn (mode 440) to the container image so sudo stopped looking up the hostname

   Lessons

   Traffic from CoreDNS to the VPC DNS resolver is limited to 1024 packets per second per interface, and drops show up only as UDP timeouts; the linklocal_allowance_exceeded counter in ethtool shows them. Names that are not fully qualified are multiplied by the search domains under ndots:5, and AAAA lookups double them again. sudo looks up the host's own name on every call unless 'Defaults !fqdn' is set.

   View case
2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

   Backpackr2025

   Symptoms

   Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.

   DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage

   Trigger

   All CoreDNS pods were scheduled onto the same node, and that node's memory usage spiked until it ran out of memory.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cachekubeletContainer memory limitNode pool

   Contributing factor

   Unreviewed defaultsSingle point of failureNo caching

   Detection

   The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.

   Remediation

   • deployed NodeLocal DNSCache as a DaemonSet in front of CoreDNS
   • added a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones for CoreDNS
   • lowered ndots from 5 to 2 with dnsConfig in the Deployments

   Lessons

   The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale. Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.

   View case
3. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets

   Cybozu2024

   Symptoms

   A team using Neco, the Kubernetes platform run by Cybozu's Cloud Platform team, reported what looked like a performance problem in the cluster DNS servers. An application in a pod that resolved cluster Services at hundreds of requests per second frequently got `no such host` and `i/o timeout` errors.

   DNS resolution failuresIntermittent timeoutsPacket loss

   Trigger

   The number of MySQL clusters watched by the monitoring application increased, and after that the name resolution errors became frequent.

   Root cause

   Capacity exhaustion

   Component

   Linux kernelClient-side DNS resolver and cache

   Contributing factor

   Unreviewed defaults

   Detection

   A team using the platform contacted the Cloud Platform team, describing the errors as a performance problem in the cluster DNS server.

   Remediation

   • Raised the kernel parameter netdev_max_backlog on the nodes
   • Set up alerts on softnet
   • Wrote a runbook for the softnet alerts

   Action items

   • Evaluate Cilium's netkit device so pod packets bypass the backlog queue

   Lessons

   Raising netdev_max_backlog makes the same failure less likely and the new alerts catch it early, but it is not a root fix: a pod that sends more packets can fill the backlog queue again. Cybozu sees replacing the veth interface with netkit, which skips the backlog queue, as the root fix.

   View case
4. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors

   MonotaRO2023

   Symptoms

   When MonotaRO raised the share of requests for the remaining endpoints of its backend API sent to EKS from 1% to 25%, Sentry filled with errors saying the API could not reach the database and outside services. The errors were about DNS name resolution, and the traffic had to be shifted back to the EC2 version.

   DNS resolution failuresHigh error rate

   Trigger

   MonotaRO raised the share of requests for the remaining API endpoints routed by the ALB to the EKS version from 1% to 25%.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cacheContainer CPU limitHorizontal pod autoscalerPod termination

   Contributing factor

   Unreviewed defaultsNo resource requests or limitsUnrepresentative stagingUnverified assumption

   Detection

   Sentry showed a large number of errors about connecting to the database and outside services right after the traffic share was raised.

   Remediation

   • Shifted the requests back to the EC2 version to stop the errors
   • Added an HPA to autoscale the CoreDNS pods as a stopgap, which let the migration to EKS finish

   Action items

   • Set ndots to 1 through dnsConfig in the application Deployments
   • Set cache 600 in the Corefile in the coredns ConfigMap to lengthen the CoreDNS cache TTL
   • Add lameduck 30s to the CoreDNS health plugin so CoreDNS keeps serving while it is scaled in
   • Consider reducing the differences between production and staging, weighed against cost

   Lessons

   MonotaRO assumed that a managed service would not cause problems and so skipped researching other companies' migrations, load testing and monitoring. Staging differed from production in its network layout, so ndots differed between them, and staging databases could not take a high-load test. Looking at the CoreDNS logs and resolv.conf instead of guessing found that ndots:5 made lookups of outside names try every search domain first.

   View case
5. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped

   Chatwork2021

   Symptoms

   In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.

   Intermittent connection failuresDNS resolution failuresIntermittent timeoutsLoad imbalance

   Trigger

   During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableCoreDNSkube-proxyClient-side DNS resolver and cacheVPC DNS resolver

   Contributing factor

   Unreviewed defaultsUnrepresentative stagingUnknown limit

   Remediation

   • Raised the conntrack maxPerCore value in the kube-proxy configuration
   • Installed dns-autoscaler so CoreDNS ran with more pods
   • Installed node-local-dns as a DaemonSet on every node
   • Replaced force_tcp with prefer_udp in the node-local-dns configuration

   Lessons

   On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.

   View case

By root cause1