Cases2 cases

1 to 2 of 2

1. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster

   Twingate2025

   Symptoms

   On 21 March 2025, between 21:10 and 21:41 UTC, part of the authentication and authorization requests to Twingate services failed with elevated error rates.

   High error rate

   Trigger

   Twingate saw anomalies in one of its GKE clusters and scaled down the deployments in that cluster to protect the rest of the system, which moved its traffic to the other clusters.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerGKE control planeHorizontal pod autoscalerCluster autoscaler

   Contributing factor

   Retry stormNo headroomOverlapping failuresHarmful manual action

   Detection

   Twingate observed anomalies in one of its GKE clusters at the start of the incident.

   Remediation

   • Reintroduced capacity in the cluster that had been scaled down, so traffic was rebalanced across regions

   Action items

   • Keep working with the cloud provider to find the cause of the unexpected timeouts and of the cluster update's impact
   • Run deployments in all regions with the same number of replicas in the HPA configuration
   • Raise the maximum node counts of the cluster node autoscalers to leave more room to scale up
   • Configure the HPA to scale up faster

   View case
2. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse

   Google Cloud2022

   Symptoms

   From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.

   High error rateHigh latencyControl plane lag

   Trigger

   An unexpected increase in backend traffic to the GKE API in us-central1.

   Root cause

   Capacity exhaustion

   Component

   GKE control plane

   Contributing factor

   Retry storm

   Remediation

   • scaled up the GKE control plane to handle the increased traffic

   Lessons

   Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.

   View case

By root cause2