Incident Library
Applied filters Root causeConfiguration changeSymptomInconsistent reads from a data storeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0560 Some PubNub messages in the US East point of presence were misrouted with no error and not delivered right away
    PubNub2026
    Symptoms

    At 12:10 UTC on 25 August 2026, PubNub's internal monitoring alerted that a very small subset of messages in one availability zone of the US East point of presence might not have been delivered right away to subscribers in the same region. Publish and Subscribe calls returned no errors, even though some messages did not appear in the responses.

    Misrouted requestsInconsistent reads from a data store
    Trigger

    During planned internal load testing, a group of internal routing servers was removed from service.

    Root cause
    Configuration change
    Component
    Service discovery
    Contributing factor
    –
    Detection

    PubNub's internal monitoring alerted at 12:10 UTC; synthetic monitoring then confirmed that no more messages were being missed.

    Remediation
    • Performed a rolling restart of the affected servers to clear the outdated routing addresses
    Action items
    • Deploy a fix to all publishing servers worldwide so that outdated internal routing addresses are retired correctly
    View case
  2. IL-0272 New Cassandra nodes served reads for data they did not hold, so Monzo records looked missing and payments failed
    Monzo2019
    Symptoms

    On 29 July 2019 at 13:14 BST, four minutes after six new servers were added to Monzo's Cassandra cluster at 13:10, automated alerts showed a small percentage of Mastercard card transactions failing, and the tool Customer Operations uses to talk to customers through in-app chat stopped working. Customers could not log in, send or receive payments or withdraw cash, and saw inaccurate balances and transactions.

    Inconsistent reads from a data storeHigh error rate
    Trigger

    At 13:10 BST Monzo added six new servers to its 21-server production Cassandra cluster to add capacity, with auto_bootstrap set to false.

    Root cause
    Configuration change
    Component
    Database
    Contributing factor
    Unverified assumptionUnrepresentative stagingMonitoring blind spotExcessive blast radiusCircular dependency
    Detection

    At 13:14 automated alerts reported a small percentage of Mastercard card transactions failing, and Customer Operations reported that its customer communication tool was not working; an incident was declared at 13:15.

    Remediation
    • Deployed a fix for an error-handling bug in a Mastercard service through a planned fallback mechanism, because the deployment tooling was failing
    • Decommissioned the six new Cassandra servers one by one so data ownership returned to the original servers
    • Reconciled the data written while the new servers were serving reads and writes, by replaying externally stored events and running consistency checks
    Action items
    • Fix the use of auto_bootstrap
    • Review and document the decisions behind the other Cassandra settings and write more extensive runbooks for operations such as scale-ups and restarts
    • Expose more Cassandra metrics and consider alerting on strong shifts in metrics such as 'row not found'
    • Split the single Cassandra cluster into several smaller clusters
    Lessons

    Monzo had gaps in its knowledge of rarer Cassandra operations such as adding servers. The scale-up test with one server could not show the problem, because quorum still came from the original replicas, so it did not match adding six servers in production. The lack of metrics pointing at Cassandra delayed the response, and a single shared cluster both let one change have a far-reaching impact and made the failing part harder to find.

    View case
Check your cluster for this

These 2 cases involve Database and Service discovery. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Database1Service discovery1
Contributing factor
Unverified assumption1Excessive blast radius1Circular dependency1Monitoring blind spot1Unrepresentative staging1
Organisation
Monzo1PubNub1
  1. IL-0560 Some PubNub messages in the US East point of presence were misrouted with no error and not delivered right away
    PubNub2026
  2. IL-0272 New Cassandra nodes served reads for data they did not hold, so Monzo records looked missing and payments failed
    Monzo2019