Cases2 cases

1 to 2 of 2

1. IL-0543 Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutes

   Twingate2025

   Symptoms

   At 20:45 UTC on 28 May 2025, most of Twingate's API traffic started going to a single backend service, which was overwhelmed and returned elevated error rates, so API calls failed widely across core services.

   Misrouted requestsLoad imbalanceHigh error rate

   Trigger

   A rollout that updated the routing rules of Twingate's global load balancer.

   Root cause

   Configuration change

   Component

   Cloud load balancer

   Contributing factor

   Unrepresentative stagingCircular dependency

   Detection

   The incident was detected at 20:48 UTC, three minutes after API traffic was first misrouted, and triage began.

   Remediation

   • Started a rollback of the routing change at 20:52 UTC; the automated rollback stalled on infrastructure dependencies
   • Applied a manual fix with local tooling at 21:09 UTC, which restored service by 21:14 UTC

   Action items

   • Correct the routing rule
   • Add alerts that detect backend traffic anomalies in lower environments
   • Improve resource scaling thresholds
   • Decouple infrastructure tooling from service dependencies so recovery has faster options
   • Introduce backend sharding to isolate customer workloads and reduce the impact of incidents

   View case
2. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN

   bunny.net2021

   Symptoms

   On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

   Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance

   Trigger

   At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

   Root cause

   Faulty deployment

   Component

   Authoritative DNS servers and recordsCDN and edge networkObject storage

   Contributing factor

   Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action

   Detection

   About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

   Remediation

   • Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
   • Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
   • Migrated the deployment systems and files to a third-party cloud storage service
   • Rewired the deployment system and DNS software to the new storage and deployed
   • Synced the databases and deployed the newest file sets through the third-party service

   Action items

   • Phase out the BinaryPack library as a hot-fix
   • Test third-party libraries more extensively
   • Migrate the internal APIs to an independent third-party service
   • Find how to stop non-critical software from becoming a single point of failure across clusters
   • Make the DNS system keep a local copy of all backup data with automatic failure detection

   Lessons

   bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

   View case

By root cause2