Cases11 cases

1 to 5 of 11

1. IL-0463 Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachable

   Harness2026

   Symptoms

   On 12 May 2026, Harness customers who used Secure Connect saw git connector tests fail with connection timeouts and CI pipelines hang or fail, while connectors without Secure Connect kept working. Continuous Integration Enterprise Mac and Linux Cloud Builds in Prod 1, Prod 2 and Prod 3 were affected.

   Network unreachableResources deleted

   Trigger

   An automated GKE control plane upgrade on the cluster that hosts Harness's internal Secure Connect FRPS (Fast Reverse Proxy Server) infrastructure.

   Root cause

   Cloud provider outage

   Component

   GKE control planeCloud controller managerCloud load balancer

   Contributing factor

   –

   Remediation

   • Ran a rollout restart of the FRPS deployment, which restored client connectivity to FRPS but not the lost IP
   • Deleted and recreated the internal service frps-internal so GKE provisioned a new load balancer and IP
   • Updated the Cloud DNS A record to the new load balancer IP

   View case
2. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours

   Wealthy2025sev1

   Symptoms

   Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

   Scale-up failureContainer crash loopService outageDNS resolution failures

   Trigger

   Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

   Root cause

   Breaking change in an upgrade

   Component

   GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets

   Contributing factor

   Deferred upgradesSingle point of failureIneffective rollbackNo headroom

   Remediation

   • Created a new GKE cluster at v1.33, the version of the forced upgrade
   • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
   • Updated the cert-manager configuration for Kubernetes 1.33
   • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
   • Decommissioned the broken cluster

   Action items

   • Configure all clusters to use the GKE stable release channel
   • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
   • Review spot instance usage for critical workloads and add on-demand fallback pools
   • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
   • Document a Kubernetes, Istio and cert-manager compatibility matrix
   • Create an automated upgrade testing pipeline
   • Reserve static IPs for quick migrations
   • Pin versions of Istio, cert-manager, CSI and DNS addons
   • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines

   Lessons

   GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

   View case
3. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster

   Twingate2025

   Symptoms

   On 21 March 2025, between 21:10 and 21:41 UTC, part of the authentication and authorization requests to Twingate services failed with elevated error rates.

   High error rate

   Trigger

   Twingate saw anomalies in one of its GKE clusters and scaled down the deployments in that cluster to protect the rest of the system, which moved its traffic to the other clusters.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerGKE control planeHorizontal pod autoscalerCluster autoscaler

   Contributing factor

   Retry stormNo headroomOverlapping failuresHarmful manual action

   Detection

   Twingate observed anomalies in one of its GKE clusters at the start of the incident.

   Remediation

   • Reintroduced capacity in the cluster that had been scaled down, so traffic was rebalanced across regions

   Action items

   • Keep working with the cloud provider to find the cause of the unexpected timeouts and of the cluster update's impact
   • Run deployments in all regions with the same number of replicas in the HPA configuration
   • Raise the maximum node counts of the cluster node autoscalers to leave more room to scale up
   • Configure the HPA to scale up faster

   View case
4. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production

   Sunwoo Park2023

   Symptoms

   On 2023-05-18 a Slack alert arrived at 20:01 and Istio stopped working properly in the production GKE cluster. The istiod error logs said that the v1beta1.Ingress and v1beta1.ValidatingWebhookConfiguration resources could not be found.

   Service outage

   Trigger

   GKE automatically upgraded the production cluster's control plane from 1.22.17-gke.7500 to 1.23.17-gke.1700 at 19:52 on 2023-05-18.

   Root cause

   Breaking change in an upgrade

   Component

   IstioGKE control planeAdmission webhook

   Contributing factor

   Deferred upgrades

   Detection

   A Slack alert at 20:01 on 2023-05-18, nine minutes after the control plane upgrade. The istiod error logs showed that the v1beta1 Ingress and ValidatingWebhookConfiguration resources could not be found.

   Remediation

   • upgraded Istio from 1.7.4 to 1.13.3

   Lessons

   The first diagnosis, end of support for Istio, was wrong; the real cause was found by matching the alert time against the cluster's UPGRADE_MASTER operations in Logs Explorer. The author recommends checking the version compatibility matrices of Kubernetes, Istio and other packages, keeping a production version for at least six months and upgrading three months before end of support, keeping the test environment on the latest version, backing up before upgrades, managing deployments and versions with a tool such as Argo CD, and testing upgrades in a test environment before production.

   View case
5. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse

   Google Cloud2022

   Symptoms

   From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.

   High error rateHigh latencyControl plane lag

   Trigger

   An unexpected increase in backend traffic to the GKE API in us-central1.

   Root cause

   Capacity exhaustion

   Component

   GKE control plane

   Contributing factor

   Retry storm

   Remediation

   • scaled up the GKE control plane to handle the increased traffic

   Lessons

   Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.

   View case

By root cause6