Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

GKE control plane

GKE 컨트롤 플레인 · gke-control-plane

7 cases · 2019 to 2025

Root cause Configuration change2 Breaking change in an upgrade2 Capacity exhaustion1 Certificate or credential expiry1 Cloud provider outage1

Contributing factor Single point of failure2 Deferred upgrades2 Unverified assumption1 Automation amplification1

Symptom Service outage5 Scale-up failure2 Control plane lag2 Control plane unresponsive2

Organisation Google Cloud1 Jetstack1 loveholidays1 Spotify1 Sunwoo Park

Cases, by severity

IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hoursWealthy · 2025 · sev1
Root cause Breaking change in an upgrade
IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in productionSunwoo Park · 2023
Root cause Breaking change in an upgrade

View on the graph

Excessive blast radius1

Ignored early signal1

Limited debug access1

Monitoring blind spot1

No health probes1

No safeguard on destructive change1

Overlapping failures1

Ineffective rollback1

Missed in review1

Unrepresentative staging1

Autoscaler misbehaviour1

Container crash loop1

Credentials unavailable1

DNS resolution failures1

High error rate1

Node recreation loop1

Resources deleted1

Stale or inconsistent API reads1

1

IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse

Google Cloud · 2022

Root cause Capacity exhaustion

IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effectWebhook Relay · 2022

Root cause Certificate or credential expiry

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

Root cause Configuration change

IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019

Root cause Configuration change

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage