Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Jobs and CronJobs

잡과 크론잡 · job-controller

3 cases · 2017 to 2025

Root cause Capacity exhaustion1 Configuration change1 Operator error1

Contributing factor Automation amplification2 Unverified assumption1 Excessive blast radius1 Unreviewed defaults1

Symptom Service outage3 Node NotReady1 Node recreation loop1 Resources deleted1

Organisation 2i2c1 Target1 Universe1

Cases, by severity

IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025
Root cause Operator error
IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018
Root cause Configuration change
IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down

View on the graph

No safeguard on destructive change1

Universe · 2017

Root cause Capacity exhaustion