Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Upgrade stall

업그레이드 정체 · upgrade-stuck

8 cases · 2019 to 2026

Root cause Faulty deployment4 Software defect2 Cloud provider outage1 Configuration change1

Component Node pool5 Node operating system image3 Cluster autoscaler2 GKE control plane2

Contributing factor Automation amplification3 Monitoring blind spot2 No health probes2 Overlapping failures2

Organisation Linode2 DFDS1 Jetstack1 loveholidays1 Microsoft1

Cases

IL-0167 A phased rollout of an LKE software release in Osaka stalled LKE-E cluster creation, version upgrades and control plane ACL changesLinode · 2026
Root cause Faulty deployment
IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025
Root cause Software defect
IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down

View on the graph

Admission webhook1

Horizontal pod autoscaler1

Ingress controller1

Kubernetes API server1

Node launch template1

Managed Kubernetes control plane and provider API1

Excessive blast radius1

Unreviewed defaults1

Ignored early signal1

Cache-masked failure1

No alert on limit1

No staged rollout1

Unrepresentative staging1

Microsoft Azure1

UK Ministry of Justice1

UK Ministry of Justice · 2025

Root cause Faulty deployment

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silentlyLinode · 2022

Root cause Faulty deployment

IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthyMicrosoft Azure · 2019

Root cause Faulty deployment

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

Root cause Configuration change

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage