IL-0542 Twingate · 2025-03-21kubernetesgcpnetworking

Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster

Twingate runs its services active-active on several Google Kubernetes Engine (GKE) clusters, with a Global Load Balancer (GLB) spreading traffic between them. When one cluster showed anomalies, Twingate scaled down its deployments, and the traffic moved mainly to a cluster that usually handles a lighter load and was sized for it. That cluster autoscaled, but the extra traffic raised error rates, retry mechanisms across services increased the load further, and a cloud provider-initiated update on the same cluster restarted pods and reduced capacity.

Twingate attributed the incident to behaviour on the cloud provider's side, namely unexpected request timeouts at the load balancer and instability during the cluster update, which together started a cascade of retry traffic. The incident ended when Twingate brought capacity back in the first cluster, so traffic was balanced across regions again and the errors and retries died down. Twingate said it was still investigating the timeout configuration and the cluster's behaviour during the update together with the provider.

Severity: –
Scope: A subset of authentication and authorization requests to Twingate services; the status page lists Authentication (Enterprise and Social), MFA, Authorization, Connector Heartbeat and management services
Users affected: –
Region: –
Time to detect: –
Time to resolve: 31m

Symptoms

On 21 March 2025, between 21:10 and 21:41 UTC, part of the authentication and authorization requests to Twingate services failed with elevated error rates.

The errors appeared on a GKE cluster that normally carries a lighter share of traffic, after deployments in another cluster were scaled down and the Global Load Balancer moved that cluster's traffic onto it. The receiving cluster started autoscaling, but the errors made services across Twingate retry their requests. Many retried requests then succeeded, but the retries also added to the total load. During the same period the cloud provider ran an update operation on that cluster, which restarted pods and took away capacity, and Twingate also saw unexpected request timeouts at the load balancer.

Symptom High error rate

Root cause

Root cause Cloud provider outage

Component Cloud load balancer GKE control plane Horizontal pod autoscaler Cluster autoscaler

Contributing factor Retry storm No headroom Overlapping failures Harmful manual action

Trigger: Twingate saw anomalies in one of its GKE clusters and scaled down the deployments in that cluster to protect the rest of the system, which moved its traffic to the other clusters.

Detection

Twingate observed anomalies in one of its GKE clusters at the start of the incident.

Remediation

Reintroduced capacity in the cluster that had been scaled down, so traffic was rebalanced across regions

Action items

Keep working with the cloud provider to find the cause of the unexpected timeouts and of the cluster update's impact
Run deployments in all regions with the same number of replicas in the HPA configuration
Raise the maximum node counts of the cluster node autoscalers to leave more room to scale up
Configure the HPA to scale up faster

Lessons

–

Sources

https://status.twingate.com/incidents/mhs89xs5rhb8 · official-status

Related cases

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019
same root cause · shared GKE control plane, Horizontal pod autoscaler, Cluster autoscaler
IL-0463 Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachableHarness · 2026
same root cause · shared Cloud load balancer, GKE control plane
IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failedANDPAD · 2026
same root cause · shared Cloud load balancer
IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)2i2c · 2022
same root cause · shared Cluster autoscaler
IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zoneUK Ministry of Justice · 2022
same root cause · shared Cloud load balancer
IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021
same root cause · shared Cluster autoscaler
IL-0446 About 20% of requests to an ITANDI ECS service got ALB 5XX errors, apparently from one zone, right after a deployITANDI · 2020
same root cause · shared Cloud load balancer
IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching usersBinder Project · 2019
same root cause · shared GKE control plane
IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hungNetlify · 2016
same root cause · shared Cloud load balancer

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer, GKE control plane, Horizontal pod autoscaler among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0542
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0542/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.