Cases2 cases

1 to 2 of 2

1. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster

   Twingate2025

   Symptoms

   On 21 March 2025, between 21:10 and 21:41 UTC, part of the authentication and authorization requests to Twingate services failed with elevated error rates.

   High error rate

   Trigger

   Twingate saw anomalies in one of its GKE clusters and scaled down the deployments in that cluster to protect the rest of the system, which moved its traffic to the other clusters.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerGKE control planeHorizontal pod autoscalerCluster autoscaler

   Contributing factor

   Retry stormNo headroomOverlapping failuresHarmful manual action

   Detection

   Twingate observed anomalies in one of its GKE clusters at the start of the incident.

   Remediation

   • Reintroduced capacity in the cluster that had been scaled down, so traffic was rebalanced across regions

   Action items

   • Keep working with the cloud provider to find the cause of the unexpected timeouts and of the cluster update's impact
   • Run deployments in all regions with the same number of replicas in the HPA configuration
   • Raise the maximum node counts of the cluster node autoscalers to leave more room to scale up
   • Configure the HPA to scale up faster

   View case
2. IL-0493 Netlify's Google Cloud TCP load balancer stopped serving about 5% of global traffic as all backends failed health checks

   Netlify2021

   Symptoms

   At 14:39 UTC on March 25, 2021, Netlify's monitors alerted that the health check of one of its load balancers was failing. That Google Cloud TCP load balancer served the single static IP that customers use for A records, and about 5% of Netlify's global traffic stopped being served until 18:46 UTC.

   Probe failuresHigh error rate

   Trigger

   Unattended operating system updates, whose timing Netlify randomises across production nodes, restarted the systemd-networkd unit on each node over several hours, and each restart removed the custom route table entry that held the load balancer IP.

   Root cause

   Software defect

   Component

   Cloud load balancersystemd-networkd

   Contributing factor

   Monitoring blind spotOverlapping failures

   Detection

   Netlify's monitors alerted at 14:39 UTC that the load balancer health check was failing. The load balancer's own backend health checks were not part of Netlify's alerting, so the backends dropping out over the preceding hours raised no alert.

   Remediation

   • Gave enterprise customers a separate range of IPs on a new load balancer for enterprise traffic
   • Provisioned an AWS network load balancer backed by Global Accelerator and published its IP for self-serve customers
   • Restarted the GCP networking daemon on all affected backend nodes, which restored traffic on the Google load balancer

   Action items

   • Patch the google-guest-agent systemd configuration on all GCP CDN nodes to react to systemd-networkd restarts
   • Report the bug to Google and open an issue in the google-guest-agent repository
   • Include the load balancer's backend health checks in alerting
   • Add a canary node to the backend to test updates and watch for issues over several days
   • Decouple traffic tiers and move to multiple regions
   • Offer customers more ways to route apex domains to Netlify, such as ALIAS and ANAME records
   • Route enterprise traffic through its own IP

   Lessons

   Netlify's own node-level health checks could not see that the backends were failing the load balancer's health check, and alerting on the load balancer's backend health checks would have given more time to diagnose before the outage.

   View case

By root cause2