IL-0543 Twingate · 2025-05-28networking

Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutes

On 28 May 2025 Twingate rolled out an update to the routing rules of its global load balancer. One new routing rule was given priority by mistake, and it sent most API traffic to a single backend. That backend service was overwhelmed and error rates rose across the API. The lower environments had not shown the problem because they carry less traffic.

The incident was detected three minutes after it began and a rollback started at 20:52 UTC. The automated rollback stalled at 21:05 UTC because infrastructure tooling it needed also depended on the affected components. At 21:09 UTC engineers applied a manual fix with local tooling, and all services had recovered by 21:14 UTC, 29 minutes after the start.

Severity: –
Scope: Control plane services (authentication, MFA, authorization, Connector Heartbeat) and management plane services (Identity Provider Sync, Public API, Admin Console, Real-Time Updates, Client Log Upload, 3rd Party Integrations, network and DNS filtering dashboards)
Users affected: –
Region: –
Time to detect: 3m
Time to resolve: 29m

Symptoms

At 20:45 UTC on 28 May 2025, most of Twingate's API traffic started going to a single backend service, which was overwhelmed and returned elevated error rates, so API calls failed widely across core services.

The failures reached the control plane services (Authentication for Enterprise and Social, Multi-Factor Authentication, Authorization, Connector Heartbeat) and the management plane services (Identity Provider Sync, Public API, Admin Console, Real-Time Updates, Client Log Upload, 3rd Party Integrations, Network Dashboards, DNS Filtering Dashboards). The incident was detected at 20:48 UTC and a rollback started at 20:52 UTC, but the automated rollback ran into delays at 21:05 UTC because infrastructure it depended on also relied on the affected components.

Symptom Misrouted requests Load imbalance High error rate

Root cause

Root cause Configuration change

Component Cloud load balancer

Contributing factor Unrepresentative staging Circular dependency

Trigger: A rollout that updated the routing rules of Twingate's global load balancer.

Detection

The incident was detected at 20:48 UTC, three minutes after API traffic was first misrouted, and triage began.

Remediation

Started a rollback of the routing change at 20:52 UTC; the automated rollback stalled on infrastructure dependencies
Applied a manual fix with local tooling at 21:09 UTC, which restored service by 21:14 UTC

Action items

Correct the routing rule
Add alerts that detect backend traffic anomalies in lower environments
Improve resource scaling thresholds
Decouple infrastructure tooling from service dependencies so recovery has faster options
Introduce backend sharding to isolate customer workloads and reduce the impact of incidents

Lessons

–

Sources

https://status.twingate.com/incidents/d57tmhvs1c5x · official-status

Related cases

IL-0346 Render's routing service returned intermittent 404s for web services and static sites in OregonRender · 2025
same root cause · shared Cloud load balancer
IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025
same root cause · shared Cloud load balancer
IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate themJenkins project · 2023
same root cause · shared Cloud load balancer
IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listenerPython Software Foundation · 2022
same root cause · shared Cloud load balancer
IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE Plus · 2022
same root cause · shared Cloud load balancer
IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022
same root cause · shared Cloud load balancer
IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021
same root cause · shared Cloud load balancer
IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021
same root cause · shared Cloud load balancer
IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster serviceDFDS · 2020
same root cause · shared Cloud load balancer
IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019
same root cause · shared Cloud load balancer

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0543
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0543/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.