Cases2 cases

1 to 2 of 2

1. IL-0543 Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutes

   Twingate2025

   Symptoms

   At 20:45 UTC on 28 May 2025, most of Twingate's API traffic started going to a single backend service, which was overwhelmed and returned elevated error rates, so API calls failed widely across core services.

   Misrouted requestsLoad imbalanceHigh error rate

   Trigger

   A rollout that updated the routing rules of Twingate's global load balancer.

   Root cause

   Configuration change

   Component

   Cloud load balancer

   Contributing factor

   Unrepresentative stagingCircular dependency

   Detection

   The incident was detected at 20:48 UTC, three minutes after API traffic was first misrouted, and triage began.

   Remediation

   • Started a rollback of the routing change at 20:52 UTC; the automated rollback stalled on infrastructure dependencies
   • Applied a manual fix with local tooling at 21:09 UTC, which restored service by 21:14 UTC

   Action items

   • Correct the routing rule
   • Add alerts that detect backend traffic anomalies in lower environments
   • Improve resource scaling thresholds
   • Decouple infrastructure tooling from service dependencies so recovery has faster options
   • Introduce backend sharding to isolate customer workloads and reduce the impact of incidents

   View case
2. IL-0532 Azure Software Load Balancer sent packets of one flow to different backends, dropping Firewall and Private Link traffic

   Microsoft Azure2022

   Symptoms

   From 05:26 UTC on 28 June 2022, some Azure customers saw intermittent network failures and packet loss when connecting to resources behind a Software Load Balancer (SLB). Telemetry showed a small increase in errors and customers raised support requests, but the failures were intermittent and were not linked to a recent deployment at first.

   Intermittent connection failuresPacket lossMisrouted requests

   Trigger

   A new SLB code deployment, rolled out through Safe Deployment Practices, made some MUX instances store the backend DIPs of load balancers with more than 15 DIPs in a different order.

   Root cause

   Faulty deployment

   Component

   Cloud load balancer

   Contributing factor

   Ignored early signal

   Detection

   Telemetry showed a small increase in errors and customers raised support requests, but the health checks of Safe Deployment Practices did not correlate these intermittent signals with the deployment until the code had reached 14 regions.

   Remediation

   • Mitigated the VIPs of customers identified as impacted with targeted manual operations
   • Built and tested a hotfix and rolled it out in phases under Safe Deployment Practices
   • Ran automation that scanned all VIPs every five minutes and fixed VIPs in an inconsistent state until the hotfix was out
   • Restored consistent ordering of the backend instances to the state before the incident

   Action items

   • Add Azure Firewall monitoring that detects when the upstream and downstream packets of one flow do not reach the same DIP
   • Keep the IPv6 flow identifier consistent for the lifetime of a connection so all its packets land on one MUX
   • Improve packet capture support for firewalls
   • Add logging of TCP flows, including invalid packets and RST, FIN and FIN-ACK packets
   • Onboard more Azure services to the Gandalf AIOps solution suite for end-to-end monitoring during safe deployments

   Lessons

   Microsoft wrote that the incident showed the need for more thorough end-to-end monitoring during safe deployments.

   View case

By root cause2