Cases2 cases

1 to 2 of 2

1. IL-0560 Some PubNub messages in the US East point of presence were misrouted with no error and not delivered right away

   PubNub2026

   Symptoms

   At 12:10 UTC on 25 August 2026, PubNub's internal monitoring alerted that a very small subset of messages in one availability zone of the US East point of presence might not have been delivered right away to subscribers in the same region. Publish and Subscribe calls returned no errors, even though some messages did not appear in the responses.

   Misrouted requestsInconsistent reads from a data store

   Trigger

   During planned internal load testing, a group of internal routing servers was removed from service.

   Root cause

   Configuration change

   Component

   Service discovery

   Contributing factor

   –

   Detection

   PubNub's internal monitoring alerted at 12:10 UTC; synthetic monitoring then confirmed that no more messages were being missed.

   Remediation

   • Performed a rolling restart of the affected servers to clear the outdated routing addresses

   Action items

   • Deploy a fix to all publishing servers worldwide so that outdated internal routing addresses are retired correctly

   View case
2. IL-0272 New Cassandra nodes served reads for data they did not hold, so Monzo records looked missing and payments failed

   Monzo2019

   Symptoms

   On 29 July 2019 at 13:14 BST, four minutes after six new servers were added to Monzo's Cassandra cluster at 13:10, automated alerts showed a small percentage of Mastercard card transactions failing, and the tool Customer Operations uses to talk to customers through in-app chat stopped working. Customers could not log in, send or receive payments or withdraw cash, and saw inaccurate balances and transactions.

   Inconsistent reads from a data storeHigh error rate

   Trigger

   At 13:10 BST Monzo added six new servers to its 21-server production Cassandra cluster to add capacity, with auto_bootstrap set to false.

   Root cause

   Configuration change

   Component

   Database

   Contributing factor

   Unverified assumptionUnrepresentative stagingMonitoring blind spotExcessive blast radiusCircular dependency

   Detection

   At 13:14 automated alerts reported a small percentage of Mastercard card transactions failing, and Customer Operations reported that its customer communication tool was not working; an incident was declared at 13:15.

   Remediation

   • Deployed a fix for an error-handling bug in a Mastercard service through a planned fallback mechanism, because the deployment tooling was failing
   • Decommissioned the six new Cassandra servers one by one so data ownership returned to the original servers
   • Reconciled the data written while the new servers were serving reads and writes, by replaying externally stored events and running consistency checks

   Action items

   • Fix the use of auto_bootstrap
   • Review and document the decisions behind the other Cassandra settings and write more extensive runbooks for operations such as scale-ups and restarts
   • Expose more Cassandra metrics and consider alerting on strong shifts in metrics such as 'row not found'
   • Split the single Cassandra cluster into several smaller clusters

   Lessons

   Monzo had gaps in its knowledge of rarer Cassandra operations such as adding servers. The scale-up test with one server could not show the problem, because quorum still came from the original replicas, so it did not match adding six servers in production. The lack of metrics pointing at Cassandra delayed the response, and a single shared cluster both let one change have a far-reaching impact and made the failing part harder to find.

   View case

By root cause1