Cases6 cases

1 to 5 of 6

1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created

   2i2c2026

   Symptoms

   At 15:10 (Bucharest time) on 2026-03-31 an alert fired for the prod hub on the berkeley-geojupyter cluster on AWS: "[FIRING:1] At least two servers failed to start in the last 30m cluster=berkeley-geojupyter hub=prod (provider=aws immediate action needed)". It opened PagerDuty incident #2057.

   Scale-up failure

   Trigger

   An upgrade merged on 2026-03-30 bumped the cluster autoscaler version, and the new version was incompatible with Kubernetes 1.33, which the berkeley-geojupyter cluster still ran.

   Root cause

   Breaking change in an upgrade

   Component

   Cluster autoscaler

   Contributing factor

   Deferred upgradesUnverified assumption

   Detection

   An alert that at least two servers failed to start in the last 30 minutes fired at 15:10 and opened PagerDuty incident #2057. The report notes it was triggered by an automatic health check run, not by users being unable to start servers.

   Remediation

   • upgraded the cluster's Kubernetes version from 1.33 to 1.34
   • downgraded the cluster autoscaler by one patch version to 9.55.0, the version meant for Kubernetes 1.34

   Action items

   • The change in https://github.com/2i2c-org/infrastructure/pull/8008 (the report gives only the link)
   • Use recurrent issues, such as the recurrent-k8s-aws-upgrades workflow, to standardize Kubernetes versions across clusters

   Lessons

   Because the context of the autoscaler bump was fresh, the team went down the right path from the start. The upgrade batches had been designed with only 1.32 clusters in mind, so the one 1.33 cluster was missed and got an incompatible autoscaler. 2i2c notes it was lucky that an automatic health check, not real users, hit the failure first.

   View case
2. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours

   Wealthy2025sev1

   Symptoms

   Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

   Scale-up failureContainer crash loopService outageDNS resolution failures

   Trigger

   Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

   Root cause

   Breaking change in an upgrade

   Component

   GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets

   Contributing factor

   Deferred upgradesSingle point of failureIneffective rollbackNo headroom

   Remediation

   • Created a new GKE cluster at v1.33, the version of the forced upgrade
   • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
   • Updated the cert-manager configuration for Kubernetes 1.33
   • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
   • Decommissioned the broken cluster

   Action items

   • Configure all clusters to use the GKE stable release channel
   • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
   • Review spot instance usage for critical workloads and add on-demand fallback pools
   • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
   • Document a Kubernetes, Istio and cert-manager compatibility matrix
   • Create an automated upgrade testing pipeline
   • Reserve static IPs for quick migrations
   • Pin versions of Istio, cert-manager, CSI and DNS addons
   • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines

   Lessons

   GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

   View case
3. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces

   ZenML2025

   Symptoms

   Between 12 AM and 8 AM CEST on 11 May 2025, every ZenML Pro workspace server was unreachable, so customers could not use workspace UIs or pipelines. The ZenML Pro dashboard was not affected.

   Service outage

   Trigger

   At 23:38 on 10 May, secrets encryption with AWS KMS was enabled through Terraform as part of the EKS module upgrade.

   Root cause

   Software defect

   Component

   TerraformNode pool

   Contributing factor

   Unrepresentative stagingDeferred upgrades

   Detection

   Engineers found the cluster degraded when they resumed the upgrade at 06:00.

   Remediation

   • deleted and recreated the EKS node group with the correct IAM permissions and configuration
   • restored service access across all affected ZenML Pro environments

   Action items

   • Upgrade all Terraform modules (EKS, VPC) to current supported versions
   • Complete the EKS cluster upgrades as planned
   • Review the strategy for periodic Kubernetes and infrastructure upgrades to prevent a recurrence

   Lessons

   A successful upgrade on a much smaller staging cluster understated the risk of the same change in production. Skipping Kubernetes and module versions let upgrade complexity build up until one upgrade carried many changes at once.

   View case
4. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production

   Sunwoo Park2023

   Symptoms

   On 2023-05-18 a Slack alert arrived at 20:01 and Istio stopped working properly in the production GKE cluster. The istiod error logs said that the v1beta1.Ingress and v1beta1.ValidatingWebhookConfiguration resources could not be found.

   Service outage

   Trigger

   GKE automatically upgraded the production cluster's control plane from 1.22.17-gke.7500 to 1.23.17-gke.1700 at 19:52 on 2023-05-18.

   Root cause

   Breaking change in an upgrade

   Component

   IstioGKE control planeAdmission webhook

   Contributing factor

   Deferred upgrades

   Detection

   A Slack alert at 20:01 on 2023-05-18, nine minutes after the control plane upgrade. The istiod error logs showed that the v1beta1 Ingress and ValidatingWebhookConfiguration resources could not be found.

   Remediation

   • upgraded Istio from 1.7.4 to 1.13.3

   Lessons

   The first diagnosis, end of support for Istio, was wrong; the real cause was found by matching the alert time against the cluster's UPGRADE_MASTER operations in Logs Explorer. The author recommends checking the version compatibility matrices of Kubernetes, Istio and other packages, keeping a production version for at least six months and upgrading three months before end of support, keeping the test environment on the latest version, backing up before upgrades, managing deployments and versions with a tool such as Argo CD, and testing upgrades in a test environment before production.

   View case
5. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes

   DFDS2022

   Symptoms

   At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

   Deploys failingContainer crash loop

   Trigger

   A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

   Root cause

   Certificate or credential expiry

   Component

   Policy engineAdmission webhook

   Contributing factor

   Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal

   Detection

   An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

   Remediation

   • reproduced the issue in a sandbox cluster and deployed the fix there first
   • updated Kyverno to a newer Helm chart version in the affected production cluster

   Action items

   • Implement monitoring on Kyverno
   • Create internal documentation on the Kyverno setup
   • Investigate the exact certificate issue in the third-party application
   • Consider changing the failurePolicy on ClusterPolicy resources going forward

   Lessons

   Operational issues should be added to the 'Ops Day to Day' board from the start.

   View case

By root cause4