IL-0204 Render · 2024-12-17 (published)kubernetesnetworking
Render runs self-managed Kubernetes clusters. In January 2022 an etcd memory spike overloaded the control plane of a cluster in Frankfurt, so deploys and cron jobs failed. The cluster's CoreDNS servers ran on the control plane nodes, and the team rebooted those nodes during the incident. CoreDNS does not need the Kubernetes API server to answer queries, since its kubernetes plugin keeps an in-memory cache, but it does need the API server to initialise. With the API server down, CoreDNS could not start again, and DNS failed for running services too.
The author notes that even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart, because the data plane was not isolated from the control plane. In response to the incident Render moved CoreDNS from control plane nodes to data plane nodes, and moved etcd to dedicated nodes because etcd performs poorly under resource contention, especially when disk write latency rises.
In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.
The incident became far more severe when DNS resolution started failing. From then on, requests to services that were already running failed as well.
Symptom Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate
Root cause Capacity exhaustion
–
–
CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running.
The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.
IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023same root cause · shared Kubernetes API server, etcd
IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019same root cause · shared Kubernetes API server, etcd
IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019same root cause · shared Kubernetes API server, etcd
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared CoreDNS
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared CoreDNS
IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025same root cause · shared CoreDNS
IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared CoreDNS
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared CoreDNS
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared CoreDNS
IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024same root cause · shared Kubernetes API server
IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agentsPalark · 2023same root cause · shared Kubernetes API server
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Kubernetes API server
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019same root cause · shared Kubernetes API server
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared CoreDNS
IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018same root cause · shared Kubernetes API server