IL-0071 Onkar Raghunath Shelke · 2026-09-17 (published)kubernetesobservability
The application image was built in one stage, so it carried the source, the build output and the whole node_modules tree including development dependencies: more than 40,000 files, over half of them in the @mui/icons-material package. containerd's content store deduplicates compressed layers by digest, but the overlayfs snapshotter unpacks every distinct layer into its own directory, so each new build with a different layer digest added another full copy of those files to every node that pulled it. The snapshots stay until the image is removed.
kubelet removes unused images only through image garbage collection, which starts when the image filesystem passes 85% of bytes and stops at 80%. It never looks at file counts. kubelet does have inode thresholds on Linux, nodefs.inodesFree and imagefs.inodesFree at 5% free, but those are hard eviction thresholds: the first action kubelet takes when inodes run low is to evict pods. Because the files were small, inodes were being used up much faster than bytes. The author works out from the df figures that the root filesystem holds about 16.3 million inodes, more than the stock mkfs ratio of one inode per 16,384 bytes allows for its size, and that it was probably formatted at about 8,192 bytes per inode; this is arithmetic, not a capture, because the author no longer has access to the cluster. Kubelet, containerd and the node image all ran on defaults.
The alert caught the trend at 67% inode use, before kubelet's eviction threshold was reached. The author's fixes are a multi-stage Dockerfile that ships only the compiled output, a flat inode usage alert alongside the predictive one, and pruning unused images on the node.
A Kubernetes worker node paged with NodeFilesystemFilesFillingUp, the node_exporter mixin alert that predicts a filesystem will run out of inodes. The engineer checked disk space first: df -h showed the root filesystem 83% used, 103G used and 22G available. df -i showed inodes 67% used, 11M used and 5.3M free.
The disk was fuller than the inode table, yet the inode table was what paged. The alert is built on predict_linear, so it fires on how fast inodes are being consumed, not on the current level; 67% was not a problem in itself, the rate of growth was.
du --inodes -xS /var showed where the files were: not in logs or pod volumes but in containerd's snapshot store, /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/. Several snapshot directories each held the same path, app/node_modules/@mui/icons-material, with 21,553 files apiece.
Symptom Node disk full
Root cause Capacity exhaustion
The NodeFilesystemFilesFillingUp alert from the node_exporter mixin fired on the predicted inode trend while inode use was at 67%.
–
kubelet's background cleanup works on bytes only. A workload heavy in small files gets no early intervention for inodes, and the first thing kubelet does at 5% free inodes is evict pods. Watch inode use on nodes separately from disk use, and keep dependency trees out of runtime images.
IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker downSaltside · 2017same root cause · shared Node root disk, kubelet, Container runtime
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared kubelet
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared kubelet
IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020same root cause · shared kubelet
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared kubelet