Cases12 cases

2 to 6 of 12

1. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes

   2i2c2025

   Symptoms

   On 12 May 2025 at 20:15 EEST a Freshdesk ticket reported that users' servers on the Earthscope hub were taking more than 15 minutes to start and that their kernels were being killed. It happened during a workshop and also affected other instructors using the hub; a second report said a short course of about 30 users was having major problems.

   Pod eviction

   Trigger

   A workshop with about 30 users on a profile option that guaranteed 8 GB of memory and set no memory limit on 32 GB nodes.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitNode pool

   Contributing factor

   No resource requests or limits

   Detection

   A Freshdesk support ticket from the community at 20:15 EEST, brought to the team's attention in the #support-freshdesk Slack channel at 20:49.

   Remediation

   • ran a helm rollout and hub redeploy, which partly improved server starts
   • made the 'medium' profile available and raised its memory guarantee to reduce overcommitment
   • pre-warmed the cluster with 15 nodes for the 30-user workshop
   • set explicit memory limits, with the guarantee equal to the limit, on the community's small-node profile options (https://github.com/2i2c-org/infrastructure/pull/6047)

   Action items

   • Write an incident report
   • Hold an internal retrospective about the incident (https://github.com/2i2c-org/meta/issues/2194)

   Lessons

   A memory guarantee without a limit lets a user server grow into memory that other pods on the same node were counting on, and on a shared node that ends with users' pods evicting each other. Setting the guarantee equal to the limit removes the overcommitment. For a known workshop, pre-warming enough nodes ahead of time keeps server start times short.

   View case
2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

   Backpackr2025

   Symptoms

   Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.

   DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage

   Trigger

   All CoreDNS pods were scheduled onto the same node, and that node's memory usage spiked until it ran out of memory.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cachekubeletContainer memory limitNode pool

   Contributing factor

   Unreviewed defaultsSingle point of failureNo caching

   Detection

   The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.

   Remediation

   • deployed NodeLocal DNSCache as a DaemonSet in front of CoreDNS
   • added a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones for CoreDNS
   • lowered ndots from 5 to 2 with dnsConfig in the Deployments

   Lessons

   The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale. Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.

   View case
3. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade

   Saifeddine Rajhi2024

   Symptoms

   After an upgrade of the kube-prometheus-stack Helm chart, several pods stayed not ready. Among them was prometheus-prometheus-operator-prometheus-0, showing 3/4 Running and a recent termination.

   OOMKilledProbe failuresMonitoring outage

   Trigger

   During the kube-prometheus-stack Helm chart upgrade, Prometheus had to replay its write-ahead log on startup, and the replay needed more memory than its limit allowed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limit

   Contributing factor

   Unknown limitNo headroom

   Detection

   The team noticed pods not ready after the Helm chart upgrade and found the OOMKilled termination with kubectl describe pod.

   Remediation

   • ran promtool tsdb analyze inside the Prometheus pod to find high-cardinality metrics such as haproxy_server_http_responses_total

   Action items

   • Verify that the memory overhead during WAL replay stays within acceptable limits, for example 10 to 15%, and do not run Prometheus close to its memory limit
   • Optimise storage, scraping and remote write configuration and upgrade to newer Prometheus releases
   • Recover automatically from OOM crash loops, for example by deleting the WAL on OOM events
   • Enforce scrape limits so Prometheus stops scraping targets when memory use passes a threshold

   Lessons

   Size Prometheus for its restart, not only for steady state: WAL replay can need far more memory than normal running, and a Prometheus that already runs near its limit can be OOM-killed during the replay. Much of Prometheus memory use comes from cardinality, which promtool tsdb analyze can measure even when Prometheus itself is down.

   View case
4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitApplication database

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
5. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady

   Scaleway2024

   Symptoms

   At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.

   Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage

   Trigger

   Simultaneous deployments in the FR-PAR region pushed the Scaleway Kubernetes API out of memory.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitkubelet

   Contributing factor

   Single point of failureRetry storm

   Detection

   Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.

   Remediation

   • raised the memory limit and garbage collection thresholds of the Kubernetes managed services
   • deployed additional replicas of key services
   • purged the registry cache and monitored the clusters as they stabilised
   • adjusted service configurations after the incident to keep the platform stable

   Action items

   • Implement a local authentication cache
   • Further develop node authentication
   • Enhance monitoring, adjust rate limiting strategies and improve incident response protocols

   Lessons

   Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.

   View case

By root cause1