Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Persistent volumes and stateful sets

영구 볼륨과 스테이트풀셋 · persistent-volume

10 cases · 2017 to 2025

Root cause Capacity exhaustion2 Configuration change2 Operator error2 Software defect2 Cloud provider outage1 Breaking change in an upgrade1

Contributing factor Unverified assumption4 Single point of failure4 No runbook3 Monitoring blind spot2

Symptom Service outage9 Container crash loop2 High error rate2 Node NotReady2

Organisation Palark2 2i2c1 AdGuard VPN1 BC Government1 Grafana Labs

Cases, by severity

IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hoursWealthy · 2025 · sev1
Root cause Breaking change in an upgrade
IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restartBC Government · 2025 · sev1
Root cause Capacity exhaustion
IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks

View on the graph

No safeguard on destructive change2

Automation amplification1

Unreviewed defaults1

Unpatched known bug1

Limited debug access1

No capacity planning1

No health probes1

No staged rollout1

Overlapping failures1

Ineffective rollback1

Missed in review1

Deferred upgrades1

Resources deleted2

Scale-up failure1

Cloud API rate limit exhausted1

DNS resolution failures1

Probe failures1

Stale or inconsistent API reads1

Stale endpoint routing1

Persistent volume full1

1

AdGuard VPN · 2025

Root cause Configuration change

IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025

Root cause Operator error

IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025

Root cause Capacity exhaustion

IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022

Root cause Configuration change

IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0Palark · 2021

Root cause Operator error

IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020

Root cause Software defect

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

Root cause Cloud provider outage

IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017

Root cause Software defect