Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Node root disk

노드 루트 디스크 · node-disk

5 cases · 2017 to 2026

Root cause Capacity exhaustion2 Cloud provider outage1 Configuration change1 Hardware failure1

Contributing factor Unreviewed defaults2 Monitoring blind spot2 Single point of failure2 Unverified assumption1

Symptom Node NotReady3 Image pull failures2 Node disk full2 Pods Pending2

Organisation Grafana Labs1 Hans H. Kim1 Onkar Raghunath Shelke1 Saltside1

Cases, by severity

IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshotsOnkar Raghunath Shelke · 2026
Root cause Capacity exhaustion
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023
Root cause Configuration change
A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joined

View on the graph

Ignored early signal1

Unpatched known bug1

No alert on limit1

No configuration tests1

Overlapping failures1

Missed in review1

Service outage2

High error rate1

IP address exhaustion1

Pods stuck in ContainerCreating1

UK Ministry of Justice1

IL-0185

Hans H. Kim · 2023

Root cause Hardware failure

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

Root cause Cloud provider outage

IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker downSaltside · 2017

Root cause Capacity exhaustion