Incident Library
Applied filters Root causeCloud provider outageSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019
    Symptoms

    On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.

    Upgrade stallPods PendingScale-up failureService outageAutoscaler misbehaviour
    Trigger

    A manual upgrade of the GKE masters and NodePools from 1.10.X to 1.12.5-gke.10, two versions at once, on 19 March 2019; GKE's release notes of 14 March had already said that version was no longer available for new clusters or master upgrades because masters showed elevated error rates, but it could still be selected.

    Root cause
    Cloud provider outage
    Component
    GKE control planeNode poolCluster autoscalerIngress controllerHorizontal pod autoscaler
    Contributing factor
    Unrepresentative stagingOverlapping failuresIgnored early signal
    Detection

    The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.

    Remediation
    • drained the remaining NodePool nodes ahead of the upgrade with kubectl drain --ignore-daemonsets --delete-local-data
    • deleted the zombie instances from the managed instance group with gcloud compute instance-groups managed delete-instances
    • cancelled the stuck UPGRADE_NODES operation with gcloud beta container operations cancel
    • used kubectl port-forward in place of the broken internal Ingresses
    • scaled down the older of two active ReplicaSets, or triggered a dummy deployment, to bring the HPA back
    Action items
    • Alert when a HorizontalPodAutoscaler is at 70% and at 100% of its capacity
    • Alert when a deployment has had two active ReplicaSets for 30 minutes and for 60 minutes
    • Re-evaluate termination grace periods that were set to arbitrary large values
    • Read the GKE release notes before every cluster upgrade
    • Use auto-upgrade in non-production environments, and manual cluster upgrades with automatic node upgrades in production
    • Keep environments as identical as possible, including resource names
    Lessons

    Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.

    View case
Check your cluster for this

These 6 cases involve Application database and Cluster autoscaler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage6 cases

Component
Application database3Cluster autoscaler2Node pool2AKS control plane1GKE control plane1Helm1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Node root disk1Persistent volumes and stateful sets1
Contributing factor
Overlapping failures3Unverified assumption2Excessive blast radius2Single point of failure2Ignored early signal1Unpatched known bug1No capacity planning1No headroom1Pages not delivered1Unrepresentative staging1
Organisation
2i2c1Devsisters1Grafana Labs1loveholidays1Microsoft Azure1Namespace1
  1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024
  2. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  3. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022
  4. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021
  5. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019
  6. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019