Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Cluster autoscaler

클러스터 오토스케일러 · cluster-autoscaler

11 cases · 2019 to 2026

Root cause Cloud provider outage3 Capacity exhaustion2 Faulty deployment2 Software defect2 Certificate or credential expiry1 Breaking change in an upgrade1

Contributing factor Unreviewed defaults3 Overlapping failures3 Unverified assumption2 No headroom2

Symptom Scale-up failure7 Pods Pending6 Service outage5 Autoscaler misbehaviour2

Organisation Linode3 2i2c2 loveholidays2 GitLab1 Hugging Face1

Cases, by severity

IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regionsLinode · 2026
Root cause Faulty deployment
IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created2i2c · 2026
Root cause Breaking change in an upgrade
Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors

View on the graph

Change with a delayed effect1

Ignored early signal1

Limited debug access1

Harmful manual action1

Monitoring blind spot1

No alert on limit1

No capacity planning1

Pages not delivered1

Ineffective rollback1

Single point of failure1

Deferred upgrades1

Unrepresentative staging1

IP address exhaustion2

Database overload1

Deploys failing1

High error rate1

UK Ministry of Justice1

IL-0156

Linode · 2025

Root cause Certificate or credential expiry

IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025

Root cause Capacity exhaustion

IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went downUK Ministry of Justice · 2025

Root cause Faulty deployment

IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clustersLinode · 2025

Root cause Software defect

IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024

Root cause Software defect

IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)2i2c · 2022

Root cause Cloud provider outage

IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021

Root cause Cloud provider outage

IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020

Root cause Capacity exhaustion

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage