Cases4 cases

1 to 4 of 4

1. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

   Backpackr2025

   Symptoms

   Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.

   DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage

   Trigger

   All CoreDNS pods were scheduled onto the same node, and that node's memory usage spiked until it ran out of memory.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cachekubeletContainer memory limitNode pool

   Contributing factor

   Unreviewed defaultsSingle point of failureNo caching

   Detection

   The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.

   Remediation

   • deployed NodeLocal DNSCache as a DaemonSet in front of CoreDNS
   • added a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones for CoreDNS
   • lowered ndots from 5 to 2 with dnsConfig in the Deployments

   Lessons

   The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale. Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.

   View case
2. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS

   Adevinta2023

   Symptoms

   On a Friday afternoon a tenant of SCHIP, Adevinta's multi-tenant Kubernetes runtime, reported a spike of HTTP 5xx errors in some of its services. The drops in quality of service came back periodically through the following days, each one short, and were reported mostly around midnight.

   High error rateDNS resolution failuresHigh latency

   Root cause

   DNS or network fault

   Component

   CoreDNSClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unreviewed defaultsUnknown limitIgnored early signalUnverified assumption

   Detection

   A tenant reported a spike of HTTP 5xx errors in some of its services; the DNS Latency SLI had already been degraded for some time without being acted on.

   Remediation

   • raised the dnsmasq concurrent query limit from 150 to 1500
   • added a CoreDNS header block that sets the ra and aa flags and clears rd, so dnsmasq caches cluster.local answers
   • configured dnsmasq on each node to answer NXDOMAIN for unresolvable search-domain names instead of forwarding them to CoreDNS
   • changed the fluent-bit log router hostname to a fully qualified name
   • asked the affected tenants to use fully qualified names for internal calls

   Lessons

   The DNS Latency SLI had shown the problem long before anyone looked at it. An SLO only protects you when its SLIs are trusted and a drop is investigated. The investigation took a month partly because several on-call groups worked on it separately, without one place for observations and hypotheses, and sometimes chased hypotheses already discarded. Disprove one hypothesis before starting the next, and collect data the metrics system lacks: DNStop showed the garbage lookups that no dashboard did. With ndots:5, short service names multiply DNS traffic; a node-local cache only helps if it can cache the cluster DNS answers at all.

   View case
3. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds

   Airbnb2019

   Symptoms

   On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.

   High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures

   Trigger

   Several services sent a high volume of external DNS queries, all of which went through kube-dns.

   Root cause

   Capacity exhaustion

   Component

   kube-dnskube-proxyClient-side DNS resolver and cache

   Contributing factor

   Unreviewed defaults

   Detection

   Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.

   Lessons

   By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.

   View case
4. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race

   MindTickle2019

   Symptoms

   Pod-to-pod calls were intermittently slow and the web application felt slow overall. A request that pod A sent at time x reached pod B at x + 5s, x + 10s or x + 15s, always a multiple of five seconds.

   Intermittent timeoutsHigh latencyDNS resolution failures

   Trigger

   The C library resolver sent the A and AAAA queries for each lookup on the same port; where DNAT or SNAT translation takes place, a race condition in conntrack sometimes left one of the queries unanswered, and the resolver waited 5 seconds before retrying.

   Root cause

   DNS or network fault

   Component

   conntrack tableClient-side DNS resolver and cacheLinux kernel

   Contributing factor

   Unreviewed defaults

   Detection

   The team noticed intermittent delays in pod-to-pod calls and slowness in the web application, and that the delays were always multiples of five seconds.

   Remediation

   • added the single-request-reopen option to dnsConfig in the deployment specs
   • set GRPC_DNS_RESOLVER=native in the Dockerfiles of gRPC applications so they used the libc resolver
   • checked with tcpdump port 53 that A and AAAA queries left from different ports

   Lessons

   Delays that are exact multiples of five seconds point to DNS lookups being retried after the resolver's 5-second timeout. A resolver option set in the pod spec only helps processes that use the libc resolver; runtimes with their own resolver, such as gRPC with c-ares, ignore it.

   View case

By root cause2