IL-0463 Harness · 2026-05-12kubernetesgcpnetworking

Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachable

Harness's Secure Connect runs on FRPS (Fast Reverse Proxy Server) in a GKE cluster, behind an internal Kubernetes LoadBalancer service. On 12 May 2026 an automated GKE control plane upgrade ran in two phases in quick succession. Each phase restarts the GKE cloud controller manager, which manages the IP addresses of LoadBalancer services and, as part of its reconciliation after a restart, deletes and re-inserts IP address reservations. The second phase began before the first reconciliation had finished, so the controller deleted the reservation for the internal FRPS load balancer and was interrupted before it could acquire it again. Harness states that none of its application deployments caused or contributed to the issue.

A rollout restart of FRPS restored client connections to the server but not the orphaned IP. Harness then deleted and recreated the internal service frps-internal, which made GKE provision a new load balancer with a new IP address, and updated the Cloud DNS A record to the new IP, restoring end-to-end routing. Connectivity was confirmed by internal tests and by affected customers. The status page shows the incident under investigation at 14:38 PDT and resolved at 16:10 PDT.

Severity: –
Scope: Secure Connect-enabled git connector tests and CI pipelines (Continuous Integration Enterprise Mac and Linux Cloud Builds in Prod 1, Prod 2 and Prod 3); connectors without Secure Connect were unaffected
Users affected: Harness customers using Secure Connect
Region: –
Time to detect: –
Time to resolve: –

Symptoms

On 12 May 2026, Harness customers who used Secure Connect saw git connector tests fail with connection timeouts and CI pipelines hang or fail, while connectors without Secure Connect kept working. Continuous Integration Enterprise Mac and Linux Cloud Builds in Prod 1, Prod 2 and Prod 3 were affected.

A load balancer that Secure Connect depends on had been left unreachable. A rollout restart of the FRPS deployment restored client connectivity to the FRPS server but did not fix the problem, because the IP address reservation of the internal load balancer was gone.

Symptom Network unreachable Resources deleted

Root cause

Root cause Cloud provider outage

Component GKE control plane Cloud controller manager Cloud load balancer

Contributing factor –

Trigger: An automated GKE control plane upgrade on the cluster that hosts Harness's internal Secure Connect FRPS (Fast Reverse Proxy Server) infrastructure.

Detection

–

Remediation

Ran a rollout restart of the FRPS deployment, which restored client connectivity to FRPS but not the lost IP
Deleted and recreated the internal service frps-internal so GKE provisioned a new load balancer and IP
Updated the Cloud DNS A record to the new load balancer IP

Action items

–

Lessons

–

Sources

https://status.harness.io/incidents/743r4kg80xj7 · official-status

Related cases

IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE clusterTwingate · 2025
same root cause · shared Cloud load balancer, GKE control plane
IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failedANDPAD · 2026
same root cause · shared Cloud load balancer
IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zoneUK Ministry of Justice · 2022
same root cause · shared Cloud load balancer
IL-0446 About 20% of requests to an ITANDI ECS service got ALB 5XX errors, apparently from one zone, right after a deployITANDI · 2020
same root cause · shared Cloud load balancer
IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019
same root cause · shared GKE control plane
IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching usersBinder Project · 2019
same root cause · shared GKE control plane
IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hungNetlify · 2016
same root cause · shared Cloud load balancer

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, GKE control plane, Cloud controller manager, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0463
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0463/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.