Cases4 cases

1 to 4 of 4

1. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout

   Cisco ThousandEyes2025

   Symptoms

   On 4 September 2025 a CoreDNS pod in a Cisco ThousandEyes Kubernetes cluster was OOMKilled.

   DNS resolution failuresStale endpoint routingOOMKilled

   Trigger

   A CoreDNS pod was OOMKilled.

   Root cause

   Software defect

   Component

   CoreDNSkube-proxyconntrack tableIstio

   Contributing factor

   Unreviewed defaults

   Lessons

   The closing slide sets three behaviours side by side: Istio's extremely aggressive DNS cache refresh with one client port reused for all DNS requests, kube-proxy's lack of a cleanup mechanism for stale conntrack entries of terminated pods (fixed in Kubernetes v1.32), and conntrack's default bidirectional UDP stream timeout of 2 minutes. The speaker describes Istio as making some very complex things simple but some very simple things complex, and presents Cilium, which looks up Services in maps instead of walking the KUBE-SERVICES chain and can replace kube-proxy entirely with its eBPF data path.

   View case
2. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB

   Lotte E-commerce2022

   Symptoms

   LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.

   High error rateIntermittent timeoutsStale endpoint routing

   Trigger

   HPA scale-in deleting pods, and the deletion of Blue pods after a blue/green switch of the Service label selector.

   Root cause

   DNS or network fault

   Component

   Horizontal pod autoscalerPod terminationkube-proxyconntrack tableCloud load balancerIngress controller

   Contributing factor

   Unreviewed defaults

   Remediation

   • delayed container shutdown on pod deletion (a wait in the SIGTERM handler or a preStop sleep, capped by terminationGracePeriodSeconds) so containers stop only after the pod has left the Service endpoints
   • switched some workloads' Ingress-created ALBs from instance mode to IP mode so traffic goes straight to pod IPs without iptables and conntrack

   Action items

   • Move all workloads to ALB IP mode gradually

   Lessons

   Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application. With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.

   View case
3. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds

   Airbnb2019

   Symptoms

   On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.

   High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures

   Trigger

   Several services sent a high volume of external DNS queries, all of which went through kube-dns.

   Root cause

   Capacity exhaustion

   Component

   kube-dnskube-proxyClient-side DNS resolver and cache

   Contributing factor

   Unreviewed defaults

   Detection

   Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.

   Lessons

   By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.

   View case
4. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods

   DevOps Hof2019

   Symptoms

   While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.

   Service outageProbe failures

   Trigger

   Cordoning and draining the old node pool removed those nodes from the GCP load balancer backend while the ingress-nginx pods were still running on them.

   Root cause

   Operator error

   Component

   Ingress controllerkube-proxyNode pool

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.

   Remediation

   • deleted the nginx-ingress-controller pods still running on the old nodes so they were rescheduled on the new, ready nodes

   Lessons

   With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.

   View case

By root cause4