Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Scale-up failure

노드 증설 실패 · autoscaler-scale-up-failed

12 cases · 2019 to 2026

Root cause Cloud provider outage3 Software defect3 Breaking change in an upgrade2 Capacity exhaustion1 Certificate or credential expiry1 Dependency failure1 Faulty deployment1

Component Node pool9 Cluster autoscaler7 GKE control plane2 Node operating system image2

Contributing factor Automation amplification2 Monitoring blind spot2 Overlapping failures2 Single point of failure2

Organisation Linode3 2i2c2 loveholidays2 Amazon Web Services1 FREE NOW

Cases

IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regionsLinode · 2026
Root cause Faulty deployment
IL-0158 An expired credential caused authentication failures that stopped LKE and LKE-E from deploying clusters and nodes in all regions, which also blocked node recycling and cluster autoscalingLinode · 2026
Root cause Certificate or credential expiry
A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created

View on the graph

Cloud load balancer1

ConfigMaps and Secrets1

Application database1

Client-side DNS resolver and cache1

Authoritative DNS servers and records1

Horizontal pod autoscaler1

Ingress controller1

Persistent volumes and stateful sets1

VPC subnet or pod address range1

Deferred upgrades2

Unverified assumption1

Unreviewed defaults1

Ignored early signal1

Cache-masked failure1

Limited debug access1

Pages not delivered1

Ineffective rollback1

Unrepresentative staging1

1

IL-0097

2i2c · 2026

Root cause Breaking change in an upgrade

IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025

Root cause Software defect

IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hoursWealthy · 2025 · sev1

Root cause Breaking change in an upgrade

IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025

Root cause Software defect

IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clustersLinode · 2025

Root cause Software defect

IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)2i2c · 2022

Root cause Cloud provider outage

IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021

Root cause Cloud provider outage

IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020

Root cause Capacity exhaustion

IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup brokeFREE NOW · 2019

Root cause Dependency failure

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage