Cases4 cases

1 to 4 of 4

1. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

   Backpackr2025

   Symptoms

   Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.

   DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage

   Trigger

   All CoreDNS pods were scheduled onto the same node, and that node's memory usage spiked until it ran out of memory.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cachekubeletContainer memory limitNode pool

   Contributing factor

   Unreviewed defaultsSingle point of failureNo caching

   Detection

   The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.

   Remediation

   • deployed NodeLocal DNSCache as a DaemonSet in front of CoreDNS
   • added a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones for CoreDNS
   • lowered ndots from 5 to 2 with dnsConfig in the Deployments

   Lessons

   The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale. Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.

   View case
2. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped

   Chatwork2021

   Symptoms

   In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.

   Intermittent connection failuresDNS resolution failuresIntermittent timeoutsLoad imbalance

   Trigger

   During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableCoreDNSkube-proxyClient-side DNS resolver and cacheVPC DNS resolver

   Contributing factor

   Unreviewed defaultsUnrepresentative stagingUnknown limit

   Remediation

   • Raised the conntrack maxPerCore value in the kube-proxy configuration
   • Installed dns-autoscaler so CoreDNS ran with more pods
   • Installed node-local-dns as a DaemonSet on every node
   • Replaced force_tcp with prefer_udp in the node-local-dns configuration

   Lessons

   On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.

   View case
3. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node

   MindTickle2020

   Symptoms

   On the Saturday the production cluster was moved from kops to EKS, the platform became very slow and too unstable to keep running, and the team reverted.

   Intermittent connection failuresHigh latency

   Trigger

   Production was moved from a kops cluster using Weave networking to a new EKS cluster using the AWS VPC CNI, without external SNAT enabled.

   Root cause

   DNS or network fault

   Component

   AWS VPC CNI

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team lead saw the platform behaving very slowly after the switch to EKS and phoned the author.

   Remediation

   • reverted production to the kops cluster
   • set AWS_VPC_K8S_CNI_EXTERNALSNAT=true in the AWS VPC CNI

   Action items

   • Monitor and alert on the AWS VPC CNI

   Lessons

   In a networking problem every component in the stack is a suspect; ruling out EKS early cost a lot of time. A DaemonSet that pinged every component from every other one exposed the anomaly that led to the fix. The team had only skimmed the AWS CNI documentation before the migration; knowing how the plugin works internally would have led to better judgements and hypotheses while debugging.

   View case
4. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver

   Toyota Connected2018

   Symptoms

   Services that used the team's Redis-dependent library crashed on start-up, and their teams reported it to the ops members. The Spring Boot logs showed the applications failing to resolve the Redis hostname during bean initialisation.

   DNS resolution failuresIntermittent connection failures

   Root cause

   Software defect

   Component

   Client-side DNS resolver and cache

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   Service teams reported start-up crashes to the ops members of the team; later the resolution failures were seen in Kibana.

   Remediation

   • set ndots to 4 with a dnsConfig block in the Kubernetes deployment manifests
   • corrected the maximum connection pool size in the Redis library, cutting Redis connections by about 70%

   Action items

   • Test Debian glibc-based base images added to the image library to see whether they resolve the DNS problems
   • Update dependencies to newer Netty versions

   Lessons

   With ndots:5, a name outside the cluster that has fewer than five dots is first tried with every cluster search domain. Adding a trailing dot avoids that, but only if every library in the path accepts it, and the Netty version in use did not. The ndots change is a workaround that will not hold forever. The author would like better alerting for this kind of failure but notes that alerts on one-off DNS failures are not wanted.

   View case

By root cause3