Incident Library

IL-0272 Monzo · 2019-07-29database

New Cassandra nodes served reads for data they did not hold, so Monzo records looked missing and payments failed

Monzo stores its data in a single Cassandra cluster of 21 servers shared by all its services, each with its own keyspace. Every piece of data is replicated to three servers chosen by its partition key, and all reads and writes use quorum, so two of the three servers must agree. The cluster had last been scaled up in October 2018, and growth in customers and microservices had brought it closer to its limits at peak load than Monzo wanted, so six servers were to be added. After the October 2018 scale-up Monzo had set auto_bootstrap to false in production, so that a failed server could be replaced from backups instead of streaming its data from the other servers. Monzo believed the flag only controlled data streaming and that new servers would join inactive until data was streamed to them one by one. The flag also decides whether a new server joins active: with it set to false, the six servers agreed on the partitions they owned and began serving queries without holding any of that data. The procedure had been tested with one new server in each of two clusters, where quorum still came from the two original replicas; with six servers in production, two or three replicas of some partitions moved to the new nodes, so some data appeared to be missing.

The first symptoms were failing card transactions and broken support tooling, and the deployment tooling failed too, so the Mastercard fix went out at 13:39 through a planned fallback mechanism. From 13:46 the team traced the internal edge's 404s to a missing key in the configuration service, then to Cassandra, and at 14:13 confirmed that a new server had answered a query. At 14:18 engineers began decommissioning the new servers one by one, each taking about 8 to 10 minutes. The 404s dropped as soon as the first was removed at 14:28, and when the last left at 15:08 Monzo worked normally again for most customers. Until 23:00 the team reconciled the data written while the new servers were serving reads and writes, by replaying externally stored events and running internal consistency checks; at 23:00 all customers could use Monzo again and the data was fully recovered. It took almost an hour from the first alert to see that Cassandra was at fault.

Severity
–
Scope
Monzo app logins, payments, ATM withdrawals, balances and transactions shown in the app, in-app chat and phone support; a small percentage of card transactions failed
Users affected
Monzo customers; most were working again at 15:08 and all at 23:00
Region
–
Time to detect
4m
Time to resolve
9h 50m

Symptoms

On 29 July 2019 at 13:14 BST, four minutes after six new servers were added to Monzo's Cassandra cluster at 13:10, automated alerts showed a small percentage of Mastercard card transactions failing, and the tool Customer Operations uses to talk to customers through in-app chat stopped working. Customers could not log in, send or receive payments or withdraw cash, and saw inaccurate balances and transactions.

At 13:29 Monzo's internal edge, the service that checks staff access and forwards requests to internal tools such as customer operations tooling and the deployment pipeline, began returning HTTP 404 responses. At 13:32 customers were being logged out of the Android and iOS apps, and at 13:33 the deployment tooling was failing as well. The internal edge was not routing internal traffic because a lookup in Monzo's configuration service, a key-value microservice that stores its data in Cassandra, returned a 404 (not found) for the key the internal edge used. Querying Cassandra directly confirmed that the key was missing, while other keys were still there. Transactions that should have existed could not be found, so account balances appeared wrong.

The evidence looked contradictory. The database and client metrics showed no increase in errors, no change in latency and normal read and write rates right after the scale-up, and the configuration service metrics showed successful reads and writes while it failed to return the key. The engineer who started the scale-up joined at 13:24, and the scale-up was ruled out as a cause because everything looked healthy. The payments team found a small error-handling bug in a Mastercard service and took it for the cause. Only at 14:13 did a query show that the answer came from one of the new servers, which the team had believed impossible.

Symptom Inconsistent reads from a data storeHigh error rate

Root cause

Root cause Configuration change

Component Database
Contributing factor Unverified assumptionUnrepresentative stagingMonitoring blind spotExcessive blast radiusCircular dependency
Trigger
At 13:10 BST Monzo added six new servers to its 21-server production Cassandra cluster to add capacity, with auto_bootstrap set to false.

Detection

At 13:14 automated alerts reported a small percentage of Mastercard card transactions failing, and Customer Operations reported that its customer communication tool was not working; an incident was declared at 13:15.

Remediation

Action items

Lessons

Monzo had gaps in its knowledge of rarer Cassandra operations such as adding servers. The scale-up test with one server could not show the problem, because quorum still came from the original replicas, so it did not match adding six servers in production. The lack of metrics pointing at Cassandra delayed the response, and a single shared cluster both let one change have a far-reaching impact and made the failing part harder to find.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.