Cases4 cases

1 to 4 of 4

1. IL-0463 Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachable

   Harness2026

   Symptoms

   On 12 May 2026, Harness customers who used Secure Connect saw git connector tests fail with connection timeouts and CI pipelines hang or fail, while connectors without Secure Connect kept working. Continuous Integration Enterprise Mac and Linux Cloud Builds in Prod 1, Prod 2 and Prod 3 were affected.

   Network unreachableResources deleted

   Trigger

   An automated GKE control plane upgrade on the cluster that hosts Harness's internal Secure Connect FRPS (Fast Reverse Proxy Server) infrastructure.

   Root cause

   Cloud provider outage

   Component

   GKE control planeCloud controller managerCloud load balancer

   Contributing factor

   –

   Remediation

   • Ran a rollout restart of the FRPS deployment, which restored client connectivity to FRPS but not the lost IP
   • Deleted and recreated the internal service frps-internal so GKE provisioned a new load balancer and IP
   • Updated the Cloud DNS A record to the new load balancer IP

   View case
2. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage

   PlanetScale2025

   Symptoms

   At 7:13 UTC on 20 October 2025, PlanetScale's continuous tests in production started failing because they could not create new database branches. The service that creates, resizes and configures database branches was unavailable, a near-total control plane outage, while existing database branches kept their capacity and connectivity.

   Service outageScale-up failureNetwork unreachableHigh error rateReplication lagDNS resolution failures

   Trigger

   The incident started with a DNS misconfiguration at one of PlanetScale's service providers. An Amazon DynamoDB outage impacted AWS STS, and several hours of capacity constraints and network instability in AWS us-east-1 followed.

   Root cause

   Cloud provider outage

   Component

   DatabaseDatabase replicationDatabase proxy and connection poolerObject storage

   Contributing factor

   –

   Detection

   PlanetScale's continuous testing in production started failing and alerted engineers at 7:13 UTC. At 10:05 UTC an alert fired for a Kubernetes operator in a large single-tenant customer installation exhausting its resources.

   Remediation

   • Disallowed new databases in us-east-1 and made us-east-2 the default region for new databases
   • Delayed new backups and cancelled pending backups that were waiting for an EC2 instance
   • Advised PlanetScale Managed customers with vtgate autoscaling to shed load
   • Paused draining and terminating EC2 instances older than 30 days
   • Kept vacant EC2 instances for reuse instead of terminating them
   • Bin-packed vtgate processes more tightly for customers with autoscaling
   • Sent manual reparent requests to move primaries to healthier availability zones
   • Restarted the edge load balancer and vtgate processes that did not recover after the partitions healed

   Action items

   • Understand and become resilient to the failure modes of SaaS dependencies for CI/CD, SSO, web application hosting and incident communication
   • Investigate more ambitious ways to reduce runtime dependence on internal and AWS services
   • Look into using all six us-east-1 availability zones to tolerate zonal outages and partitions

   Lessons

   Keeping the control plane separate from the data plane meant the control plane outage did not affect customer databases, and redundancy, automated failover and zonal traffic routing limited the effect of the network partitions. Network partitions are hard to reason about, test and tolerate: with three availability zones a system survives losing one zone only if the other two can still reach each other reliably.

   View case
3. IL-0261 Discord went fully offline twice as short network losses on Google Cloud sessions nodes wedged Erlang distribution

   Discord2017

   Symptoms

   On 17 and 18 November 2017 nodes of Discord's 'sessions' cluster on Google Cloud Platform lost network connectivity for short periods, 16 times in all. Engineers saw a service impact, a node whose service had died and restarted, and pages for low memory on other sessions nodes; twice the failure cascaded and Discord was completely offline for about 30 minutes.

   Network unreachableMemory pressureService outage

   Trigger

   A software change that Google rolled out to the Google Cloud Platform networking stack on Friday introduced a bug that made VMs in Discord's sessions cluster lose network connectivity for short periods.

   Root cause

   Cloud provider outage

   Component

   Service discovery

   Contributing factor

   Limited debug accessHarmful manual action

   Detection

   At 11:52 PST on 17 November engineers noticed a service impact and escalated it in the incident chat. Later affected nodes showed up as pages to the on-call engineer for low memory on sessions nodes, and on 18 November as a page for a networking incident on sessions-1-11 together with issue reports that the customer experience team escalated from support channels.

   Remediation

   • Forcefully terminated sessions nodes whose Erlang distribution port had failed
   • Escalated the Google support ticket to P1 and notified the Technical Account Manager
   • Restarted several core Discord services on 17 November, making clients reconnect over about 20 minutes
   • Began terminating and rebooting the entire sessions cluster after seeing that rebooted machines did not fail again
   • Disabled message sending globally on 18 November to help the service recover
   • Restarted all members of the sessions cluster
   • Re-enabled message sending after the initial all clear
   • Google rolled back the software change made to its networking stack

   Action items

   • Continue the project to improve how Discord's Erlang clusters handle node failure
   • Expand how upgrades to these services are load tested so the rollout can be completed safely
   • Replace the HTTP library used by Discord's Elixir/Erlang services
   • Fix the instability seen in Erlang's net_kernel component
   • Replace the pieces of the service discovery component that degraded with more performant implementations
   • Work with Google on the issues Discord met while working with Google on this incident
   • Expand the internal procedure for communicating with Google so the progress of P1 issues is tracked

   Lessons

   Discord concluded that performance and stability issues in its own system, in Erlang's net_kernel, the service discovery component and the HTTP library, were what turned short network losses on part of the fleet from isolated issues for some users into two full system outages.

   View case
4. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung

   Netlify2016

   Symptoms

   At 8am PST on Monday, August 22, 2016, Netlify got alerts that its central load balancer, the single public IP that users point A records at, had stopped responding. Rackspace had null-routed the load balancer, so all traffic to that IP hung before it reached Netlify's servers.

   Network unreachable

   Trigger

   A DDoS attack against Netlify's central load balancer in Rackspace's Chicago datacenter, after which Rackspace null-routed the load balancer's public IP.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancer

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts at 8am PST reported that the central load balancer had stopped responding, and Rackspace then confirmed it had null-routed the load balancer.

   Remediation

   • Started a new cluster of CDN machines in Google Compute behind a Google network load balancer
   • Gave clients who had contacted support the new public IP and told them to change their A records
   • Updated the documentation and announced the new IP on Twitter, Gitter and the status page

   Action items

   • Keep the new Google-based network IP running in place of the Rackspace load balancer
   • Release Netlify's own DNS hosting, integrated with the CDN, so domains do not depend on a fixed IP
   • Fix the status page publishing process so it is updated straight away during an issue
   • Build a mailer that sends targeted mails to all paying clients during an outage

   Lessons

   Netlify judged Rackspace's load balancer too vulnerable to DDoS and Rackspace's DDoS processes not up to the standard of other cloud providers, and expected Google's anycast-based Maglev load balancer to be more resilient. In the end, the only way to protect a domain from DDoS-related outages is not to expose any fixed IP address for it, which is why sites should use the www domain with a CNAME, or Netlify DNS, rather than a single A-record IP.

   View case

By root cause1