Cases2 cases

1 to 2 of 2

1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours

   Wealthy2025sev1

   Symptoms

   Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

   Scale-up failureContainer crash loopService outageDNS resolution failures

   Trigger

   Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

   Root cause

   Breaking change in an upgrade

   Component

   GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets

   Contributing factor

   Deferred upgradesSingle point of failureIneffective rollbackNo headroom

   Remediation

   • Created a new GKE cluster at v1.33, the version of the forced upgrade
   • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
   • Updated the cert-manager configuration for Kubernetes 1.33
   • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
   • Decommissioned the broken cluster

   Action items

   • Configure all clusters to use the GKE stable release channel
   • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
   • Review spot instance usage for critical workloads and add on-demand fallback pools
   • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
   • Document a Kubernetes, Istio and cert-manager compatibility matrix
   • Create an automated upgrade testing pipeline
   • Reserve static IPs for quick migrations
   • Pin versions of Istio, cert-manager, CSI and DNS addons
   • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines

   Lessons

   GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

   View case
2. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade

   Gunosy2023

   Symptoms

   During a traffic surge from a breaking news alert, pods of a Gunosy service on Kubernetes could not start because the containers could not read their settings from the ConfigMap. The pods that kept running took the extra load and were restarted, failed to start again, and pod after pod died.

   Container crash loop

   Trigger

   A breaking news alert increased traffic and pods were scaled out automatically while a release that removed an old database's settings from the ConfigMap had been rolled out to only one canary pod.

   Root cause

   Faulty deployment

   Component

   ConfigMaps and SecretsHelmProgressive delivery controller

   Contributing factor

   Partial rolloutAutomation amplification

   Action items

   • Configure Argo Rollouts to roll out automatically, with setWeight 1, a one-hour pause and then setWeight 100
   • Add the "helm.sh/resource-policy": keep annotation so Helm keeps past versions of the ConfigMap
   • Build and use a Helm plugin, helm-clear, that keeps only the last few ConfigMap versions by the revision number in their names, set to keep 5 in CD

   Lessons

   Gunosy holds that a review that only resolves to be careful next time would soon be followed by the same failure, because human attention cannot be relied on, so each incident gets a permanent fix built into the system. Keeping past ConfigMap versions also made it possible to roll back quickly without going through CI/CD, which a ConfigMap change had previously prevented.

   View case

By root cause2