Cases2 cases

1 to 2 of 2

1. IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicas

   Razorpay2020

   Symptoms

   On a Friday evening in December 2019, Razorpay's internal monitoring alerted on an unusually high number of 5xx errors. The SRE team found that the main application, called API, could not connect to its RDS MySQL database, but within about 110 seconds the application recovered by itself and the alerts stopped. The RDS logs showed that the master instance had gone through a Multi-AZ failover.

   High error rateReplication lagData loss

   Trigger

   The master instance of the API application's RDS MySQL database went through a Multi-AZ failover to its standby in another availability zone.

   Root cause

   Configuration change

   Component

   DatabaseDatabase replicationDatabase backups and snapshots

   Contributing factor

   –

   Detection

   An internal monitoring alert on unusually high 5xx errors, then alerts that replication had failed on all replicas. The data loss was found the next day by matching the database against the application trace logs.

   Remediation

   • Moved all read replica traffic to the master instance
   • Recreated all five read replicas after a six-hour full snapshot and moved traffic back to them
   • Recovered the missing queries from the binary logs and corrected the data stored in RDS
   • Changed innodb_flush_log_at_trx_commit on the master from 2 to 1

   Lessons

   With innodb_flush_log_at_trx_commit set to 2, Razorpay had been open to data loss on any failover all along, even though RDS Multi-AZ copies everything on the primary's disk to the standby. The incident also taught the team how RDS runs a Multi-AZ setup and its failovers.

   View case
2. IL-0215 Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedly

   Twilio2013

   Symptoms

   At 2:39 AM PDT (9:39 UTC) on 18 July 2013, services that relied on the redis-master of Twilio's billing system began to fail because the load on its host had become extreme. At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster.

   Database overloadData lossWrites rejected

   Trigger

   At 1:35 AM PDT on 18 July a loss of network connectivity disconnected all billing redis-slaves from the redis-master at once, and all of them requested a full synchronization at the same time. On-call engineers then restarted the overloaded redis-master.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Retry stormHarmful manual actionUnsafe failure behaviour

   Detection

   At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster. At 3:28 AM PDT the billing system monitoring reported the anomalous charges, and at 12:36 PM PDT monitoring detected the recurrence.

   Remediation

   • Restarted the redis-master to relieve the high load, which loaded a wrong configuration
   • Took the billing system offline to stop further credit card charges
   • Restored service to all suspended accounts
   • Restored the lost account balances from the independent double-bookkeeping record in the relational datastore
   • Took billing offline again after the recurrence and reactivated the newly suspended accounts
   • Refunded all erroneous credit card charges
   • Gave all affected accounts a credit of 10% of their last 30 days of spend
   • Replaced the original Redis cluster and corrected the redis-master configuration
   • Activated the billing system progressively for groups of customer accounts

   Action items

   • Disable Redis restarts on the redis-master and recover the master by pivoting a slave instead
   • Add fail-safes so the billing system neither suspends accounts nor charges cards when balances are missing or cannot be written
   • Validate the billing system against the double-bookkeeping databases in real time

   Lessons

   Losing the in-flight balances and the ability to write balances at the same time exposed a flaw in the auto-recharge system: it failed in a dangerous way, charging cards and suspending accounts instead of stopping.

   View case

By root cause1