Cases6 cases

1 to 5 of 6

1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots

   Onkar Raghunath Shelke2026

   Symptoms

   A Kubernetes worker node paged with NodeFilesystemFilesFillingUp, the node_exporter mixin alert that predicts a filesystem will run out of inodes. The engineer checked disk space first: df -h showed the root filesystem 83% used, 103G used and 22G available. df -i showed inodes 67% used, 11M used and 5.3M free.

   Node disk full

   Trigger

   Several versions of a single-stage Node.js image, each shipping more than 40,000 files, were held on the node; containerd unpacked every layer with a distinct digest into its own snapshot, each with a full copy of node_modules.

   Root cause

   Capacity exhaustion

   Component

   Node root diskkubeletContainer runtime

   Contributing factor

   Unreviewed defaultsMonitoring blind spotUnknown limit

   Detection

   The NodeFilesystemFilesFillingUp alert from the node_exporter mixin fired on the predicted inode trend while inode use was at 67%.

   Action items

   • Build the image in multiple stages so the final image contains only the compiled assets
   • Add a flat alert on root filesystem inode use above 80% for 30 minutes alongside the predictive alert
   • Put a Prometheus query on a dashboard that shows, per node, inode usage minus disk usage, sorted from the highest
   • Remove unused images from nodes with crictl rmi --prune to free old snapshots

   Lessons

   kubelet's background cleanup works on bytes only. A workload heavy in small files gets no early intervention for inodes, and the first thing kubelet does at 5% free inodes is evict pods. Watch inode use on nodes separately from disk use, and keep dependency trees out of runtime images.

   View case
2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

   Backpackr2025

   Symptoms

   Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.

   DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage

   Trigger

   All CoreDNS pods were scheduled onto the same node, and that node's memory usage spiked until it ran out of memory.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cachekubeletContainer memory limitNode pool

   Contributing factor

   Unreviewed defaultsSingle point of failureNo caching

   Detection

   The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.

   Remediation

   • deployed NodeLocal DNSCache as a DaemonSet in front of CoreDNS
   • added a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones for CoreDNS
   • lowered ndots from 5 to 2 with dnsConfig in the Deployments

   Lessons

   The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale. Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.

   View case
3. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady

   Scaleway2024

   Symptoms

   At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.

   Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage

   Trigger

   Simultaneous deployments in the FR-PAR region pushed the Scaleway Kubernetes API out of memory.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitkubelet

   Contributing factor

   Single point of failureRetry storm

   Detection

   Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.

   Remediation

   • raised the memory limit and garbage collection thresholds of the Kubernetes managed services
   • deployed additional replicas of key services
   • purged the registry cache and monitored the clusters as they stabilised
   • adjusted service configurations after the incident to keep the platform stable

   Action items

   • Implement a local authentication cache
   • Further develop node authentication
   • Enhance monitoring, adjust rate limiting strategies and improve incident response protocols

   Lessons

   Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.

   View case
4. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999

   LINE2020

   Symptoms

   One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.

   Container crash loopOOMKilledMonitoring outageProbe failures

   Trigger

   After start, the Prometheus time series count rose to 2,162,256 within five minutes and its memory use filled the 8GB host.

   Root cause

   Capacity exhaustion

   Component

   PrometheuskubeletLinux kernel

   Contributing factor

   No resource requests or limitsNo capacity planning

   Detection

   The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.

   Remediation

   • removed most of the unnecessary settings from the Prometheus ConfigMap, after which the container stopped crashing

   Action items

   • Set appropriate resource requests and limits on the Prometheus pod
   • Reduce unneeded metrics by adjusting scrape targets or scrape interval
   • Estimate Prometheus memory needs, provide dedicated hosts with a node label such as deploy-topology: Prometheus, and pin the Prometheus pod to them with a nodeSelector

   Lessons

   Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.

   View case
5. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit

   Zalando2019

   Symptoms

   Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

   High error rateCredentials unavailableControl plane lag

   Trigger

   An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

   Root cause

   Capacity exhaustion

   Component

   kubeletCredential broker for podsKubernetes API serverCI system

   Contributing factor

   Unreviewed defaultsSingle point of failure

   Remediation

   • Scaled the CDP cluster up by hand so builder pods spread over more than one node

   Action items

   • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration

   View case

By root cause1