Cases2 cases

1 to 2 of 2

1. IL-0463 Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachable

   Harness2026

   Symptoms

   On 12 May 2026, Harness customers who used Secure Connect saw git connector tests fail with connection timeouts and CI pipelines hang or fail, while connectors without Secure Connect kept working. Continuous Integration Enterprise Mac and Linux Cloud Builds in Prod 1, Prod 2 and Prod 3 were affected.

   Network unreachableResources deleted

   Trigger

   An automated GKE control plane upgrade on the cluster that hosts Harness's internal Secure Connect FRPS (Fast Reverse Proxy Server) infrastructure.

   Root cause

   Cloud provider outage

   Component

   GKE control planeCloud controller managerCloud load balancer

   Contributing factor

   –

   Remediation

   • Ran a rollout restart of the FRPS deployment, which restored client connectivity to FRPS but not the lost IP
   • Deleted and recreated the internal service frps-internal so GKE provisioned a new load balancer and IP
   • Updated the Cloud DNS A record to the new load balancer IP

   View case
2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours

   Magic Pages2025

   Symptoms

   At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

   Service outageNode NotReadyCloud API rate limit exhausted

   Trigger

   One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

   Root cause

   Capacity exhaustion

   Component

   Cloud controller managerPersistent volumes and stateful sets

   Contributing factor

   Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook

   Detection

   The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

   Remediation

   • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
   • waited until ratelimit-remaining had built up a buffer before starting recovery
   • restarted the control plane servers
   • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
   • restarted the remaining websites by hand

   Action items

   • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
   • Get a higher Hetzner API rate limit; Hetzner support was contacted
   • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
   • Document and test a procedure for recovering Longhorn after a full cluster shutdown

   Lessons

   The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

   View case

By root cause2