IL-0027 Backpackr · 2025-05-07 (published)kubernetesawsnetworking
Backpackr runs the idus handmade marketplace and the Tumblbug crowdfunding platform on Amazon EKS. CoreDNS on the production cluster still had the setup it got when the cluster was created: two pods, kept on the on-demand node group with a nodeSelector, and no other tuning. Every pod also used the Kubernetes default ndots:5 in /etc/resolv.conf, so any name with fewer than five dots was first tried with each search domain appended. A lookup of idus.com went out as up to five queries (idus.com.idus-backends.svc.cluster.local, idus.com.svc.cluster.local, idus.com.cluster.local, idus.com.ap-northeast-2.compute.internal, and finally idus.com), and only the last one could succeed. Under production traffic this multiplied the load on CoreDNS, which showed up as slow or failed lookups and intermittent connection failures to RDS and Redis. The problem was much more visible in production than in the dev and stage environments.
The outage came when both CoreDNS pods ended up on one node. The pods had no memory limit and ran in the BestEffort QoS class. When the node's memory usage spiked, the node ran out of memory, the kubelet stopped, and all pods on the node fell into CrashLoopBackOff. Because there was no CoreDNS pod on any other node, the cluster lost DNS and the service went down completely.
Backpackr fixed it by putting NodeLocal DNSCache in front of CoreDNS as a DaemonSet that caches answers on every node, by spreading CoreDNS with a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones (both are now EKS defaults; older EKS versions did not set them), and by lowering ndots from 5 to 2 through dnsConfig in the Deployments. Appending a trailing dot to every domain name was rejected because every reference in code and configuration would have had to change. Istio DNS proxying was tested but not adopted because it overlaps with NodeLocal DNSCache. The write-up also suggests running CoreDNS on its own node group or on Fargate and using affinity and priorityClassName.
Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.
The team first checked application code, node resources, pod status and container settings. A closer look showed that the failing calls were failed DNS resolutions. Tuning the CoreDNS cache and its bufsize setting eased the symptoms for a while but did not remove them.
The worst episode was a full outage. At one point every CoreDNS pod was running on the same on-demand node. That node's memory usage jumped until the node ran out of memory, the kubelet stopped working, and every pod on the node went into CrashLoopBackOff. With no CoreDNS pod left to answer, DNS stopped working and the whole service went down.
Symptom DNS resolution failuresIntermittent connection failuresHigh latency
The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.
–
The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale.
Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Node pool, Container memory limit, CoreDNS
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit, Client-side DNS resolver and cache, CoreDNS
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared Node pool, CoreDNS
IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025same root cause · shared Client-side DNS resolver and cache, CoreDNS
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node pool, CoreDNS
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Node pool, Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit, kubelet
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Node pool, Container memory limit
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Node pool, Container memory limit
IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshotsOnkar Raghunath Shelke · 2026same root cause · shared kubelet
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared CoreDNS
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared CoreDNS
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Node pool
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020same root cause · shared kubelet
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool
IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019same root cause · shared Client-side DNS resolver and cache
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019same root cause · shared kubelet