Cases5 cases

1 to 5 of 5

1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots

   Onkar Raghunath Shelke2026

   Symptoms

   A Kubernetes worker node paged with NodeFilesystemFilesFillingUp, the node_exporter mixin alert that predicts a filesystem will run out of inodes. The engineer checked disk space first: df -h showed the root filesystem 83% used, 103G used and 22G available. df -i showed inodes 67% used, 11M used and 5.3M free.

   Node disk full

   Trigger

   Several versions of a single-stage Node.js image, each shipping more than 40,000 files, were held on the node; containerd unpacked every layer with a distinct digest into its own snapshot, each with a full copy of node_modules.

   Root cause

   Capacity exhaustion

   Component

   Node root diskkubeletContainer runtime

   Contributing factor

   Unreviewed defaultsMonitoring blind spotUnknown limit

   Detection

   The NodeFilesystemFilesFillingUp alert from the node_exporter mixin fired on the predicted inode trend while inode use was at 67%.

   Action items

   • Build the image in multiple stages so the final image contains only the compiled assets
   • Add a flat alert on root filesystem inode use above 80% for 30 minutes alongside the predictive alert
   • Put a Prometheus query on a dashboard that shows, per node, inode usage minus disk usage, sorted from the highest
   • Remove unused images from nodes with crictl rmi --prune to free old snapshots

   Lessons

   kubelet's background cleanup works on bytes only. A workload heavy in small files gets no early intervention for inodes, and the first thing kubelet does at 5% free inodes is evict pods. Watch inode use on nodes separately from disk use, and keep dependency trees out of runtime images.

   View case
2. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device

   UK Ministry of Justice2023

   Symptoms

   At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

   Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion

   Trigger

   Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

   Root cause

   Configuration change

   Component

   Node root diskNode launch templateNode poolAWS VPC CNICoreDNS

   Contributing factor

   Missed in reviewNo configuration testsNo alert on limit

   Detection

   The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

   Remediation

   • Uncordoned the old default node group and drained the new nodes to move workloads back
   • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
   • Drained the old node group onto the fixed nodes

   Action items

   • Update the runbook to compare launch template changes during an EKS module upgrade
   • Create a test setup that pulls images of different sizes, like live
   • Update the RootVolUtilisation alert runbook to check the disk space configuration
   • Scale CoreDNS dynamically based on the number of nodes
   • Investigate whether IPv6 can solve the IP prefix starvation problem
   • Add drift testing to identify when a terraform plan shows a change to the launch template
   • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation

   Lessons

   The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

   View case
3. IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joined

   Hans H. Kim2023

   Symptoms

   The kernel on a master node of the Kubernetes cluster logged disk I/O errors: "[HAWKEYE] kernel:: [22329228.540857] blk_update_request: I/O error, dev sda, sector 795030104 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0". The sda disk switched to read-only, and the machine needed a new disk and an operating system reinstall.

   Node NotReady

   Trigger

   A hardware failure of the disk on the first control plane node.

   Root cause

   Hardware failure

   Component

   KubesprayetcdNode root diskConfigMaps and Secrets

   Contributing factor

   Single point of failure

   Detection

   Disk I/O errors from blk_update_request on dev sda appeared in the node's kernel log.

   Remediation

   • reordered the kube_control_plane group in the Kubespray inventory so that node-2 became the first control plane node
   • removed the failed machine from the cluster
   • edited the cluster-info ConfigMap in kube-public to point at a live control plane node and updated certificate-authority-data
   • installed and configured kubelet and the other components on a new node and joined it as a control plane node with kubeadm, skipping the etcd join phase
   • added the etcd member separately

   Lessons

   In a Kubespray cluster the first node of the kube_control_plane inventory group is special, so its failure needs its own procedure: move another node into the first position, fix the cluster-info ConfigMap, join a new control plane node and add the etcd member back. etcd should be brought back to full membership before another control plane node fails.

   View case
4. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage

   Grafana Labs2019

   Symptoms

   An alert fired because some Metrictank instances could not drain their write queue, which meant they could not write to Cassandra. Google had not yet published an incident, but a team member's Discord server, also hosted on GCP, was down, which pointed at GCP; Google confirmed an SSD persistent disk incident about 90 minutes after it began.

   High error rateNode NotReadyPods PendingService outage

   Trigger

   A Google Cloud incident in which SSD persistent disks performed so poorly that many I/O operations failed for an extended period.

   Root cause

   Cloud provider outage

   Component

   Persistent volumes and stateful setsApplication databaseNode root disk

   Contributing factor

   Overlapping failuresUnpatched known bugUnverified assumption

   Detection

   An alert that Metrictank write instances could not flush their write queue to Cassandra. Error rates alone rose too slowly to notice quickly, because most queries were served from memory.

   Remediation

   • Added compute nodes hoping new nodes could attach the disks, which did not work
   • Deleted and recreated the Cassandra StatefulSet with an updated pod management policy, so all pods but one started
   • Raised Kafka topic retention to avoid data loss during the long downtime
   • Temporarily added nodes so Metrictank restarts were not slowed by CPU starvation
   • Restarted all Metrictank instances, which took more than 20 hours, so they connected to the new Cassandra IPs

   Action items

   • Re-instantiate the gocql connection in Metrictank when it is detected dead, so it reconnects even when all Cassandra IPs change
   • Fix the Metrictank write-queue deadlock by updating an internal data structure with atomics instead of a lock

   Lessons

   An OrderedReady StatefulSet cannot recover past a single pod whose volume will not attach; when every pod is down, the ordering that protects normal start-up blocks the whole cluster. A client that learns peer addresses only from live peers needs a way to go back to the service name when all peers are replaced at once. A long-tolerated bug with a config workaround can surface at the worst time; this one turned out to have a simple fix. Writing incoming data to a Kafka write-ahead log first meant all data could still be ingested after recovery, and Kafka on non-SSD disks escaped the disk incident. With both Metrictank bugs fixed, the team expects a similar event to take less than 3 hours to recover from instead of 23.

   View case
5. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down

   Saltside2017

   Symptoms

   While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

   Pods PendingImage pull failuresNode NotReadyService outage

   Trigger

   A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

   Root cause

   Capacity exhaustion

   Component

   Node root diskkubeletContainer runtimeHelm

   Contributing factor

   Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure

   Detection

   The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

   Remediation

   • terminated the three worker nodes and let the auto scaling group replace them

   Action items

   • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
   • Increase the worker root EBS volumes from 20GB to 120GB
   • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
   • Configure different allocation limits for the different layers in Kubernetes
   • Discuss a highly available Tiller with the Helm team
   • Find a way to throttle the number of newly created pods

   Lessons

   An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

   View case

By root cause4