IL-0182 이언철 · 2024-04-16 (published)kubernetesaws
The author, a DevOps engineer, describes CoreDNS problems on an Amazon EKS cluster that uses Karpenter. Under fast-changing traffic, too few CoreDNS pods handled the DNS load: latency rose, the network dropped intermittently for task pods, and the horizontal pod autoscaler's scale-out drove a surge in database connections that spread the impact across the service.
The first round of fixes sized CoreDNS to its capacity and made it scale with the cluster. The author notes that each CoreDNS pod needs a certain amount of resources and that average latency and queries per second depend on how much each pod can handle. The team set up the Cluster Proportional Autoscaler (CPA), which scales CoreDNS by cluster size, choosing nodesPerReplica and coresPerReplica for the instance types in use and tuning the replica minimum and maximum through monitoring. They spread CoreDNS with topologySpreadConstraints instead of anti-affinity, as Karpenter suggests, using the soft ScheduleAnyway setting so Karpenter could still consolidate and replace nodes for drift; EKS add-on versions v1.9.3-eksbuild.11 and v1.10.1-eksbuild.7 and later include topologySpreadConstraints by default. They also set alert thresholds on CoreDNS error rate, panics, request duration and request rate, and in New Relic used a Prometheus agent to integrate kube-dns with an agent status dashboard.
The over-provisioned CoreDNS pods then piled up 3 to 4 per node on the few nodes they could run on, and those nodes could not be consolidated. To take load off the main CoreDNS, the team deployed NodeLocal DNSCache, a CoreDNS DaemonSet on every node that answers from its cache or forwards to the main CoreDNS. DNS query volume to the main CoreDNS fell sharply, spikes became easier to absorb, and a DNS problem now stays on one node instead of spreading to the whole service. After this change the production metrics improved considerably and the node over-expansion stopped. The author concludes that CPA together with NodeLocal DNSCache resolved the CoreDNS load problem.
When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.
Task pods, which start when a request comes in, ran on nodes that come and go. The small number of CoreDNS pods could not keep up with them, so the network dropped intermittently and these pods did not work properly.
After the first fixes, a second problem appeared. CoreDNS replicas had been raised well beyond need for stability. Every node carried a taint, so the CoreDNS Deployment's pods landed only on particular nodes (for example the default ones), and to keep topologySpreadConstraints maxSkew: 1 the scheduler placed 3 to 4 CoreDNS pods per node. Those nodes could not be consolidated, and the node count grew too large.
Symptom High latencyIntermittent connection failuresAutoscaler misbehaviour
–
–
A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared CoreDNS, Horizontal pod autoscaler
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared CoreDNS
IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed RunningKevin · 2026same root cause · shared Horizontal pod autoscaler
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Pod scheduling and placement
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
IL-0048same root cause · shared CoreDNS
IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared CoreDNS
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared CoreDNS
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared CoreDNS
IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 spaceAdevinta · 2024same root cause · shared Karpenter
IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Pod scheduling and placement
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared CoreDNS
IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017same root cause · shared Pod scheduling and placement