Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
    Symptoms

    At 15:10 (Bucharest time) on 2026-03-31 an alert fired for the prod hub on the berkeley-geojupyter cluster on AWS: "[FIRING:1] At least two servers failed to start in the last 30m cluster=berkeley-geojupyter hub=prod (provider=aws immediate action needed)". It opened PagerDuty incident #2057.

    Scale-up failure
    Trigger

    An upgrade merged on 2026-03-30 bumped the cluster autoscaler version, and the new version was incompatible with Kubernetes 1.33, which the berkeley-geojupyter cluster still ran.

    Root cause
    Breaking change in an upgrade
    Component
    Cluster autoscaler
    Contributing factor
    Deferred upgradesUnverified assumption
    Detection

    An alert that at least two servers failed to start in the last 30 minutes fired at 15:10 and opened PagerDuty incident #2057. The report notes it was triggered by an automatic health check run, not by users being unable to start servers.

    Remediation
    • upgraded the cluster's Kubernetes version from 1.33 to 1.34
    • downgraded the cluster autoscaler by one patch version to 9.55.0, the version meant for Kubernetes 1.34
    Action items
    • The change in https://github.com/2i2c-org/infrastructure/pull/8008 (the report gives only the link)
    • Use recurrent issues, such as the recurrent-k8s-aws-upgrades workflow, to standardize Kubernetes versions across clusters
    Lessons

    Because the context of the autoscaler bump was fresh, the team went down the right path from the start. The upgrade batches had been designed with only 1.32 clusters in mind, so the one 1.33 cluster was missed and got an incompatible autoscaler. 2i2c notes it was lucky that an automatic health check, not real users, hit the failure first.

    View case
  2. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025sev1
    Symptoms

    Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

    Scale-up failureContainer crash loopService outageDNS resolution failures
    Trigger

    Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

    Root cause
    Breaking change in an upgrade
    Component
    GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets
    Contributing factor
    Deferred upgradesSingle point of failureIneffective rollbackNo headroom
    Remediation
    • Created a new GKE cluster at v1.33, the version of the forced upgrade
    • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
    • Updated the cert-manager configuration for Kubernetes 1.33
    • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
    • Decommissioned the broken cluster
    Action items
    • Configure all clusters to use the GKE stable release channel
    • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
    • Review spot instance usage for critical workloads and add on-demand fallback pools
    • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
    • Document a Kubernetes, Istio and cert-manager compatibility matrix
    • Create an automated upgrade testing pipeline
    • Reserve static IPs for quick migrations
    • Pin versions of Istio, cert-manager, CSI and DNS addons
    • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines
    Lessons

    GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

    View case
  3. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
    Symptoms

    On 2023-05-18 a Slack alert arrived at 20:01 and Istio stopped working properly in the production GKE cluster. The istiod error logs said that the v1beta1.Ingress and v1beta1.ValidatingWebhookConfiguration resources could not be found.

    Service outage
    Trigger

    GKE automatically upgraded the production cluster's control plane from 1.22.17-gke.7500 to 1.23.17-gke.1700 at 19:52 on 2023-05-18.

    Root cause
    Breaking change in an upgrade
    Component
    IstioGKE control planeAdmission webhook
    Contributing factor
    Deferred upgrades
    Detection

    A Slack alert at 20:01 on 2023-05-18, nine minutes after the control plane upgrade. The istiod error logs showed that the v1beta1 Ingress and ValidatingWebhookConfiguration resources could not be found.

    Remediation
    • upgraded Istio from 1.7.4 to 1.13.3
    Lessons

    The first diagnosis, end of support for Istio, was wrong; the real cause was found by matching the alert time against the cluster's UPGRADE_MASTER operations in Logs Explorer. The author recommends checking the version compatibility matrices of Kubernetes, Istio and other packages, keeping a production version for at least six months and upgrading three months before end of support, keeping the test environment on the latest version, backing up before upgrades, managing deployments and versions with a tool such as Argo CD, and testing upgrades in a test environment before production.

    View case
Check your cluster for this

These 3 cases involve GKE control plane and Admission webhook. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade3 cases

Component
GKE control plane2Admission webhook1Cluster autoscaler1ConfigMaps and Secrets1Client-side DNS resolver and cache1Istio1Node pool1Persistent volumes and stateful sets1
Contributing factor
Deferred upgrades3Unverified assumption1No headroom1Ineffective rollback1Single point of failure1
Organisation
2i2c1Sunwoo Park1Wealthy1
  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
  2. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  3. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023