Cases2 cases

1 to 2 of 2

1. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster

   Twingate2025

   Symptoms

   On 21 March 2025, between 21:10 and 21:41 UTC, part of the authentication and authorization requests to Twingate services failed with elevated error rates.

   High error rate

   Trigger

   Twingate saw anomalies in one of its GKE clusters and scaled down the deployments in that cluster to protect the rest of the system, which moved its traffic to the other clusters.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerGKE control planeHorizontal pod autoscalerCluster autoscaler

   Contributing factor

   Retry stormNo headroomOverlapping failuresHarmful manual action

   Detection

   Twingate observed anomalies in one of its GKE clusters at the start of the incident.

   Remediation

   • Reintroduced capacity in the cluster that had been scaled down, so traffic was rebalanced across regions

   Action items

   • Keep working with the cloud provider to find the cause of the unexpected timeouts and of the cluster update's impact
   • Run deployments in all regions with the same number of replicas in the HPA configuration
   • Raise the maximum node counts of the cluster node autoscalers to leave more room to scale up
   • Configure the HPA to scale up faster

   View case
2. IL-0261 Discord went fully offline twice as short network losses on Google Cloud sessions nodes wedged Erlang distribution

   Discord2017

   Symptoms

   On 17 and 18 November 2017 nodes of Discord's 'sessions' cluster on Google Cloud Platform lost network connectivity for short periods, 16 times in all. Engineers saw a service impact, a node whose service had died and restarted, and pages for low memory on other sessions nodes; twice the failure cascaded and Discord was completely offline for about 30 minutes.

   Network unreachableMemory pressureService outage

   Trigger

   A software change that Google rolled out to the Google Cloud Platform networking stack on Friday introduced a bug that made VMs in Discord's sessions cluster lose network connectivity for short periods.

   Root cause

   Cloud provider outage

   Component

   Service discovery

   Contributing factor

   Limited debug accessHarmful manual action

   Detection

   At 11:52 PST on 17 November engineers noticed a service impact and escalated it in the incident chat. Later affected nodes showed up as pages to the on-call engineer for low memory on sessions nodes, and on 18 November as a page for a networking incident on sessions-1-11 together with issue reports that the customer experience team escalated from support channels.

   Remediation

   • Forcefully terminated sessions nodes whose Erlang distribution port had failed
   • Escalated the Google support ticket to P1 and notified the Technical Account Manager
   • Restarted several core Discord services on 17 November, making clients reconnect over about 20 minutes
   • Began terminating and rebooting the entire sessions cluster after seeing that rebooted machines did not fail again
   • Disabled message sending globally on 18 November to help the service recover
   • Restarted all members of the sessions cluster
   • Re-enabled message sending after the initial all clear
   • Google rolled back the software change made to its networking stack

   Action items

   • Continue the project to improve how Discord's Erlang clusters handle node failure
   • Expand how upgrades to these services are load tested so the rollout can be completed safely
   • Replace the HTTP library used by Discord's Elixir/Erlang services
   • Fix the instability seen in Erlang's net_kernel component
   • Replace the pieces of the service discovery component that degraded with more performant implementations
   • Work with Google on the issues Discord met while working with Google on this incident
   • Expand the internal procedure for communicating with Google so the progress of P1 issues is tracked

   Lessons

   Discord concluded that performance and stability issues in its own system, in Erlang's net_kernel, the service discovery component and the HTTP library, were what turned short network losses on part of the fleet from isolated issues for some users into two full system outages.

   View case

By root cause1