Cases2 cases

1 to 2 of 2

1. IL-0463 Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachable

   Harness2026

   Symptoms

   On 12 May 2026, Harness customers who used Secure Connect saw git connector tests fail with connection timeouts and CI pipelines hang or fail, while connectors without Secure Connect kept working. Continuous Integration Enterprise Mac and Linux Cloud Builds in Prod 1, Prod 2 and Prod 3 were affected.

   Network unreachableResources deleted

   Trigger

   An automated GKE control plane upgrade on the cluster that hosts Harness's internal Secure Connect FRPS (Fast Reverse Proxy Server) infrastructure.

   Root cause

   Cloud provider outage

   Component

   GKE control planeCloud controller managerCloud load balancer

   Contributing factor

   –

   Remediation

   • Ran a rollout restart of the FRPS deployment, which restored client connectivity to FRPS but not the lost IP
   • Deleted and recreated the internal service frps-internal so GKE provisioned a new load balancer and IP
   • Updated the Cloud DNS A record to the new load balancer IP

   View case
2. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung

   Netlify2016

   Symptoms

   At 8am PST on Monday, August 22, 2016, Netlify got alerts that its central load balancer, the single public IP that users point A records at, had stopped responding. Rackspace had null-routed the load balancer, so all traffic to that IP hung before it reached Netlify's servers.

   Network unreachable

   Trigger

   A DDoS attack against Netlify's central load balancer in Rackspace's Chicago datacenter, after which Rackspace null-routed the load balancer's public IP.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancer

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts at 8am PST reported that the central load balancer had stopped responding, and Rackspace then confirmed it had null-routed the load balancer.

   Remediation

   • Started a new cluster of CDN machines in Google Compute behind a Google network load balancer
   • Gave clients who had contacted support the new public IP and told them to change their A records
   • Updated the documentation and announced the new IP on Twitter, Gitter and the status page

   Action items

   • Keep the new Google-based network IP running in place of the Rackspace load balancer
   • Release Netlify's own DNS hosting, integrated with the CDN, so domains do not depend on a fixed IP
   • Fix the status page publishing process so it is updated straight away during an issue
   • Build a mailer that sends targeted mails to all paying clients during an outage

   Lessons

   Netlify judged Rackspace's load balancer too vulnerable to DDoS and Rackspace's DDoS processes not up to the standard of other cloud providers, and expected Google's anycast-based Maglev load balancer to be more resilient. In the end, the only way to protect a domain from DDoS-related outages is not to expose any fixed IP address for it, which is why sites should use the www domain with a CNAME, or Netlify DNS, rather than a single A-record IP.

   View case

By root cause1