Incident Library
Applied filters ComponentNode poolSymptomPods PendingClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019
    Symptoms

    On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.

    Upgrade stallPods PendingScale-up failureService outageAutoscaler misbehaviour
    Trigger

    A manual upgrade of the GKE masters and NodePools from 1.10.X to 1.12.5-gke.10, two versions at once, on 19 March 2019; GKE's release notes of 14 March had already said that version was no longer available for new clusters or master upgrades because masters showed elevated error rates, but it could still be selected.

    Root cause
    Cloud provider outage
    Component
    GKE control planeNode poolCluster autoscalerIngress controllerHorizontal pod autoscaler
    Contributing factor
    Unrepresentative stagingOverlapping failuresIgnored early signal
    Detection

    The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.

    Remediation
    • drained the remaining NodePool nodes ahead of the upgrade with kubectl drain --ignore-daemonsets --delete-local-data
    • deleted the zombie instances from the managed instance group with gcloud compute instance-groups managed delete-instances
    • cancelled the stuck UPGRADE_NODES operation with gcloud beta container operations cancel
    • used kubectl port-forward in place of the broken internal Ingresses
    • scaled down the older of two active ReplicaSets, or triggered a dummy deployment, to bring the HPA back
    Action items
    • Alert when a HorizontalPodAutoscaler is at 70% and at 100% of its capacity
    • Alert when a deployment has had two active ReplicaSets for 30 minutes and for 60 minutes
    • Re-evaluate termination grace periods that were set to arbitrary large values
    • Read the GKE release notes before every cluster upgrade
    • Use auto-upgrade in non-production environments, and manual cluster upgrades with automatic node upgrades in production
    • Keep environments as identical as possible, including resource names
    Lessons

    Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.

    View case
Check your cluster for this

These 8 cases involve Node pool and Cluster autoscaler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Faulty deployment1 case

Component
Cluster autoscalerDeschedulerNode pool
Contributing factor
Unreviewed defaultsNo alert on limitNo headroom
Organisation
UK Ministry of Justice
  1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025

Configuration change1 case

Component
Node poolAWS VPC CNI
Contributing factor
Overlapping failuresUnpinned version
Organisation
CoW Protocol
  1. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024

Capacity exhaustion2 cases

Component
Node pool2Cluster autoscaler1Prometheus1VPC subnet or pod address range1
Contributing factor
Excessive blast radius1Unreviewed defaults1Unknown limit1Monitoring blind spot1No headroom1
Organisation
2i2c1loveholidays1
  1. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
  2. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020

Cloud provider outage3 cases

Component
Cluster autoscaler3Node pool3GKE control plane1Horizontal pod autoscaler1Ingress controller1
Contributing factor
Overlapping failures2Ignored early signal1Limited debug access1Monitoring blind spot1Pages not delivered1Single point of failure1Unrepresentative staging1
Organisation
2i2c1GitLab1loveholidays1
  1. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022
  2. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021
  3. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019

Dependency failure1 case

Component
Node operating system imageNode pool
Contributing factor
Monitoring blind spot
Organisation
FREE NOW
  1. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke
    FREE NOW2019