Cases2 cases

1 to 2 of 2

1. IL-0543 Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutes

   Twingate2025

   Symptoms

   At 20:45 UTC on 28 May 2025, most of Twingate's API traffic started going to a single backend service, which was overwhelmed and returned elevated error rates, so API calls failed widely across core services.

   Misrouted requestsLoad imbalanceHigh error rate

   Trigger

   A rollout that updated the routing rules of Twingate's global load balancer.

   Root cause

   Configuration change

   Component

   Cloud load balancer

   Contributing factor

   Unrepresentative stagingCircular dependency

   Detection

   The incident was detected at 20:48 UTC, three minutes after API traffic was first misrouted, and triage began.

   Remediation

   • Started a rollback of the routing change at 20:52 UTC; the automated rollback stalled on infrastructure dependencies
   • Applied a manual fix with local tooling at 21:09 UTC, which restored service by 21:14 UTC

   Action items

   • Correct the routing rule
   • Add alerts that detect backend traffic anomalies in lower environments
   • Improve resource scaling thresholds
   • Decouple infrastructure tooling from service dependencies so recovery has faster options
   • Introduce backend sharding to isolate customer workloads and reduce the impact of incidents

   View case
2. IL-0272 New Cassandra nodes served reads for data they did not hold, so Monzo records looked missing and payments failed

   Monzo2019

   Symptoms

   On 29 July 2019 at 13:14 BST, four minutes after six new servers were added to Monzo's Cassandra cluster at 13:10, automated alerts showed a small percentage of Mastercard card transactions failing, and the tool Customer Operations uses to talk to customers through in-app chat stopped working. Customers could not log in, send or receive payments or withdraw cash, and saw inaccurate balances and transactions.

   Inconsistent reads from a data storeHigh error rate

   Trigger

   At 13:10 BST Monzo added six new servers to its 21-server production Cassandra cluster to add capacity, with auto_bootstrap set to false.

   Root cause

   Configuration change

   Component

   Database

   Contributing factor

   Unverified assumptionUnrepresentative stagingMonitoring blind spotExcessive blast radiusCircular dependency

   Detection

   At 13:14 automated alerts reported a small percentage of Mastercard card transactions failing, and Customer Operations reported that its customer communication tool was not working; an incident was declared at 13:15.

   Remediation

   • Deployed a fix for an error-handling bug in a Mastercard service through a planned fallback mechanism, because the deployment tooling was failing
   • Decommissioned the six new Cassandra servers one by one so data ownership returned to the original servers
   • Reconciled the data written while the new servers were serving reads and writes, by replaying externally stored events and running consistency checks

   Action items

   • Fix the use of auto_bootstrap
   • Review and document the decisions behind the other Cassandra settings and write more extensive runbooks for operations such as scale-ups and restarts
   • Expose more Cassandra metrics and consider alerting on strong shifts in metrics such as 'row not found'
   • Split the single Cassandra cluster into several smaller clusters

   Lessons

   Monzo had gaps in its knowledge of rarer Cassandra operations such as adding servers. The scale-up test with one server could not show the problem, because quorum still came from the original replicas, so it did not match adding six servers in production. The lack of metrics pointing at Cassandra delayed the response, and a single shared cluster both let one change have a far-reaching impact and made the failing part harder to find.

   View case

By root cause1