Why does CoreDNS get OOMKilled and take down DNS for the whole cluster?

Why one pod's limit becomes everyone's outage

CoreDNS is the DNS server that every pod in a Kubernetes cluster uses to find services and outside names. It usually runs as a Deployment of two pods behind the kube-dns Service. If those pods go over their memory limit, the kubelet kills them with exit code 137, OOMKilled, and until a replacement is serving, name lookups fail for every application in the cluster, and often for the monitoring that would have raised the alarm.

The Zalando postmortem from January 2019 is the case most people cite. Their CoreDNS pods had a memory limit of 100Mi, set from historical usage. When retries in an aggregation layer turned into a burst of DNS queries, every CoreDNS pod went over the limit and was killed at once, and each restarted pod was killed again under the continuing load. Alerting ran inside the cluster and needed DNS, so nobody was paged automatically; the on-call engineer raised the limit by hand to 2000Mi and the cluster recovered.

What multiplies the queries

Zalando named three multipliers. The application used the default Node.js resolver, which caches no lookups. It ran with the default Kubernetes DNS configuration, where ndots:5 means that any name with fewer than five dots is first tried with every search domain appended, so one lookup of an external name became about ten queries. And its HTTP client kept connections short-lived, so it resolved names again and again.

Backpackr, running two marketplaces on EKS, saw the same ndots effect in production in 2025: a lookup of idus.com went out as up to five queries, only the last of which could succeed, and the extra load showed up as slow lookups and intermittent connection failures to their databases and caches. Lowering ndots from 5 to 2 through dnsConfig in their Deployments was one of three changes they made.

Too few replicas, or all of them on one node

Backpackr's outage itself came when both CoreDNS pods ended up on the same node. They had no memory limit, so they ran in the BestEffort class; when the node ran out of memory the kubelet stopped and every pod on it crashed, and with no CoreDNS pod anywhere else the cluster lost DNS entirely. They added a PodDisruptionBudget and topology spread constraints across zones, which newer EKS versions now set by default, and put NodeLocal DNSCache on every node in front of CoreDNS.

RebelMouse lost one CoreDNS pod when a planned node reroll for security patches terminated its instance in May 2025. Its share of queries fell on the remaining pods, which had been sized at three times their average load and still could not take the spike; DNS slowed and timed out across many services for about eight minutes until the pod was rescheduled and the replica count raised.

After the kill: stale connection tracking

Losing a CoreDNS pod can hurt even after a replacement is running. DNS uses UDP, and the Linux conntrack table remembers which backend pod a client's queries were sent to. Preply's February 2020 postmortem describes their CoreDNS autoscaler scaling from three pods to two, kube-proxy failing to delete the conntrack entries for the removed pod, and some services' queries still going to an address with no CoreDNS behind it for 26 minutes.

A 2025 SREcon talk by a Cisco ThousandEyes engineer traces a similar chain after a CoreDNS pod was OOMKilled: Istio's sidecar refreshes its DNS cache aggressively from one reused client port, kube-proxy had no mechanism to clean up conntrack entries of terminated pods until Kubernetes v1.32 fixed issue 126130, and conntrack's default timeout for a UDP stream is two minutes. Three ordinary behaviours, together, kept queries flowing to a dead pod.

What to check in your cluster

1. Look at the CoreDNS pods in kube-system: restart counts, the last termination reason, and whether the memory limit was set from a quiet period. OOMKilled in kubectl describe is the direct sign.
2. Read the ndots and search settings in a pod's /etc/resolv.conf and count how many queries one external lookup produces. Lowering ndots in dnsConfig, or using fully qualified names with a trailing dot, cuts the load at the source; Backpackr rejected the trailing dot because every reference would have had to change.
3. Check where the CoreDNS pods are running and whether a PodDisruptionBudget and spread constraints keep them apart. Two pods on one node is one node away from a total outage.
4. Check that your alerting can reach you when cluster DNS is down. Zalando's response was slow because the paging path itself needed DNS.

The changes the five teams made

Zalando raised the memory limit twentyfold on the spot and concluded that the blast radius of one application's DNS traffic should be limited to a node or a pod, not the cluster. Backpackr added NodeLocal DNSCache, spread constraints with a disruption budget, and ndots 2. RebelMouse raised the replica count so the load spread across more pods. Preply reverted a release and let a deploy rewrite the conntrack table, and admitted the fix treated consequences, not the root cause. The ThousandEyes talk ends by presenting an eBPF data path that does not walk the kube-proxy chains at all.

Incident Library holds these five cases and the other DNS cases in the library, each with a link to the original report; the search accepts a sentence such as DNS lookups failing after CoreDNS was OOMKilled and returns them grouped by root cause. An AI assistant connected to the MCP server at https://incidentlib-mcp.benchlet.com/mcp gets the same cases with their sources, and a script can call GET https://incidentlib-api.benchlet.com/search?q=coredns oomkilled dns outage.

Frequently asked questions