Cases24 cases

4 to 8 of 24

1. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout

   Cisco ThousandEyes2025

   Symptoms

   On 4 September 2025 a CoreDNS pod in a Cisco ThousandEyes Kubernetes cluster was OOMKilled.

   DNS resolution failuresStale endpoint routingOOMKilled

   Trigger

   A CoreDNS pod was OOMKilled.

   Root cause

   Software defect

   Component

   CoreDNSkube-proxyconntrack tableIstio

   Contributing factor

   Unreviewed defaults

   Lessons

   The closing slide sets three behaviours side by side: Istio's extremely aggressive DNS cache refresh with one client port reused for all DNS requests, kube-proxy's lack of a cleanup mechanism for stale conntrack entries of terminated pods (fixed in Kubernetes v1.32), and conntrack's default bidirectional UDP stream timeout of 2 minutes. The speaker describes Istio as making some very complex things simple but some very simple things complex, and presents Cilium, which looks up Services in maps instead of walking the KUBE-SERVICES chain and can replace kube-proxy entirely with its eBPF data path.

   View case
2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

   Backpackr2025

   Symptoms

   Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.

   DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage

   Trigger

   All CoreDNS pods were scheduled onto the same node, and that node's memory usage spiked until it ran out of memory.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cachekubeletContainer memory limitNode pool

   Contributing factor

   Unreviewed defaultsSingle point of failureNo caching

   Detection

   The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.

   Remediation

   • deployed NodeLocal DNSCache as a DaemonSet in front of CoreDNS
   • added a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones for CoreDNS
   • lowered ndots from 5 to 2 with dnsConfig in the Deployments

   Lessons

   The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale. Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.

   View case
3. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade

   Saifeddine Rajhi2024

   Symptoms

   After an upgrade of the kube-prometheus-stack Helm chart, several pods stayed not ready. Among them was prometheus-prometheus-operator-prometheus-0, showing 3/4 Running and a recent termination.

   OOMKilledProbe failuresMonitoring outage

   Trigger

   During the kube-prometheus-stack Helm chart upgrade, Prometheus had to replay its write-ahead log on startup, and the replay needed more memory than its limit allowed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limit

   Contributing factor

   Unknown limitNo headroom

   Detection

   The team noticed pods not ready after the Helm chart upgrade and found the OOMKilled termination with kubectl describe pod.

   Remediation

   • ran promtool tsdb analyze inside the Prometheus pod to find high-cardinality metrics such as haproxy_server_http_responses_total

   Action items

   • Verify that the memory overhead during WAL replay stays within acceptable limits, for example 10 to 15%, and do not run Prometheus close to its memory limit
   • Optimise storage, scraping and remote write configuration and upgrade to newer Prometheus releases
   • Recover automatically from OOM crash loops, for example by deleting the WAL on OOM events
   • Enforce scrape limits so Prometheus stops scraping targets when memory use passes a threshold

   Lessons

   Size Prometheus for its restart, not only for steady state: WAL replay can need far more memory than normal running, and a Prometheus that already runs near its limit can be OOM-killed during the replay. Much of Prometheus memory use comes from cardinality, which promtool tsdb analyze can measure even when Prometheus itself is down.

   View case
4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitDatabase

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
5. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR

   Scaleway2024

   Symptoms

   At 01:04 CET on 13 February 2024, the Kubernetes API behind Scaleway's managed Kubernetes services Kapsule and Kosmos in the Paris region (FR-PAR) spiked in RAM and CPU use and ran out of memory. Nodes in FR-PAR could no longer authenticate to their control plane, and some of them went "Not Ready". Customers on both shared (mutualized) and dedicated environments were affected.

   Node NotReadyControl plane lagOOMKilled

   Trigger

   A new Kubernetes release was made available to all users on 12 February 2024, and during the night of 12 to 13 February many clusters triggered their auto-upgrade at the same time.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API server

   Contributing factor

   Retry stormAutomation amplificationNo staged rollout

   Detection

   The timeline's first entry, at 01:04 CET, records the Kubernetes API's RAM and CPU spikes and out-of-memory state, with nodes unable to authenticate to their control plane and some becoming "Not Ready". The first recorded action, 42 minutes later at 01:46, was to increase the resources allocated to the API.

   Remediation

   • increased the resources allocated to the Kubernetes API at 01:46 and again at 02:20
   • applied API traffic restrictions at 02:30 to smooth the incoming traffic
   • removed the traffic restrictions progressively in 50-100 batches from 03:20
   • ran a script at 03:40 to replace the failing nodes created during the congestion
   • rolled back to the former node authentication mechanism, first on a subset of clusters from 15:45 and on all clusters by 19:41 on 19 February
   • disabled cluster upgrade and configuration update features until 10:30 on 20 February

   Action items

   • Increase the resources allocated to the Kubernetes API
   • Roll back to the former node authentication mechanism
   • Implement a local authentication cache on nodes
   • Rework the resource allocation strategy to behave better under load spikes
   • Add new request queuing mechanisms for Kubernetes
   • Separate the different components to reduce side effects under high load or failure
   • Monitor selected metrics closely to identify anomalous patterns earlier
   • Refund customers who deployed nodes during the incident but could not use them normally

   Lessons

   Letting many clusters auto-upgrade at the same moment after a release turns routine upgrades into one coordinated load spike on the shared Kubernetes API. A node authentication mechanism that retries without letting up turns a slow API into an unavailable one, and auto-healing that replaces NotReady nodes amplifies the problem. The congestion pattern had already appeared three weeks earlier, and short-term mitigations did not stop it from recurring, so Scaleway rolled back the new authentication mechanism. Scaleway notes that as the number of customer clusters and nodes grows sharply, traffic control mechanisms have to improve to prevent congestion.

   View case

By root cause5