IL-0082 RebelMouse · 2025-05-19kubernetesawsnetworking
CoreDNS is the DNS server that services inside a Kubernetes cluster use to find each other. During a planned reroll of the EKS nodes for security patches, the EC2 instance running one CoreDNS pod was terminated. Its share of DNS queries fell on the remaining CoreDNS pods, which could not handle the extra load, so DNS resolution slowed down and timed out for many services.
The CoreDNS deployment had been given CPU and memory of at least three times its average load. The root cause analysis found that this headroom still did not cover the sudden spike on the remaining pods when one pod disappeared. The team rescheduled the lost CoreDNS pod onto a new EC2 instance and raised the replica count of the CoreDNS deployment so the load spread evenly across more pods. Core functionality was back at 13:50 UTC, eight minutes after detection, and the incident was resolved at 14:13 UTC.
At 13:42 UTC on 19 May 2025, RebelMouse's monitoring tools reported a higher error rate and degraded performance across services. A routine reroll of the EKS nodes had started 27 minutes earlier, at 13:15 UTC.
DNS lookups through CoreDNS became slow and timed out more often. Multiple services and applications were disrupted, and because this happened during peak usage hours, users widely reported the platform as unavailable.
Symptom DNS resolution failuresHigh latencyHigh error rateService outage
Root cause Capacity exhaustion
Monitoring tools detected an increased error rate and degraded service performance at 13:42 UTC, and the tech team started investigating at 13:44 UTC.
–
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Node pool, CoreDNS
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared Node pool, CoreDNS
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Node pool, CoreDNS
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared CoreDNS
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Node pool
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared CoreDNS
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared CoreDNS
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Node pool
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Node pool
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared CoreDNS