Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Node recreation loop

노드 재생성 반복 · nodes-recreated-in-loop

3 cases · 2017 to 2025

Root cause Capacity exhaustion1 Configuration change1 Software defect1

Component Admission webhook1 GKE control plane1 GPU driver1 Jobs and CronJobs1

Contributing factor Automation amplification3 Unverified assumption2 Excessive blast radius2 Monitoring blind spot1

Organisation Inflab1 Jetstack1 Universe1

Cases

IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025
Root cause Software defect
IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019
Root cause Configuration change
IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down

View on the graph

Kubernetes API server1

Node operating system image1

Pod scheduling and placement1

No health probes1

Universe · 2017

Root cause Capacity exhaustion