IL-0001 Zalando · 2019-01-07kubernetesaws
The CoreDNS pods in the cluster had a memory limit of 100Mi, set from historical resource usage. When a retry spike at the aggregation layer turned into a burst of DNS queries, every CoreDNS pod went over that limit and was OOM-killed at once. Under the continuing load each restarted pod was killed again, and nothing could raise the limit automatically: Zalando had been running the Vertical Pod Autoscaler for some system components for a few months, but not for CoreDNS. The whole cluster lost DNS.
Several properties of the aggregation layer multiplied its DNS queries. It is a Node.js application using the default Node.js DNS resolution, which caches no lookups. It ran with the default Kubernetes DNS configuration, so the "ndots 5 problem" turned every lookup of a cluster-external name into 10 DNS queries. Its HTTP client connection pool had a short maximum connection lifetime, chosen because some dependencies switch traffic through DNS, so it kept opening new connections and resolving names again.
Most alerting ran inside the cluster and needed DNS, so nobody was paged automatically and the response was slow and manual. Once the Kubernetes on-call engineer was reached by phone, they raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi. CoreDNS recovered and handled the load, the aggregation layer closed its circuit breakers, and the Fashion Store recovered completely.
Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.
It started when calls from the aggregation layer to one of its many downstream services began to time out. The aggregation layer answered those timed-out calls with 404 responses, clients retried, and the number of requests to the aggregation layer spiked. The CoreDNS queries per second in the cluster rose by the same amount, and the memory usage of the CoreDNS pods climbed until all of them ran out of memory and were OOM-killed at the same moment.
The CoreDNS pods did not recover. Each restarted pod met the same constant query load, ran out of memory again and was killed again, so DNS was down for the entire cluster. The aggregation layer could no longer resolve any hostname, and because it had no DNS cache it opened its circuit breakers to its downstream services.
The cluster's internal monitoring went down at the same time, because pushing metrics and triggering alerts needs to reach external services. No automatic page went out. The on-call engineer for the aggregation layer had to phone the on-call Kubernetes engineer, who then found the CoreDNS pods being OOM-killed.
Root cause Capacity exhaustion
The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.
The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods.
Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Client-side DNS resolver and cache, Container memory limit, CoreDNS
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Container memory limit, CoreDNS
IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025same root cause · shared Client-side DNS resolver and cache, CoreDNS
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared CoreDNS
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared CoreDNS
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Container memory limit
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared CoreDNS
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared CoreDNS
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Container memory limit
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Container memory limit
IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019same root cause · shared Client-side DNS resolver and cache
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0005 linked by the editor