Cases2 cases

1 to 2 of 2

1. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat

   37signals2020

   Symptoms

   At 11:59AM CDT on Friday 28 August 2020, Basecamp's Ops team received a flood of alerts from PagerDuty, Nagios and Prometheus. Monitoring showed that the newer of the two metro links between the data centers in Ashburn, VA and Chicago, IL had been lost for about 20-30 seconds, and BC3 kept responding slowly after the link came back.

   Link flappingHigh latencyReplication lag

   Trigger

   Both redundant metro links between Basecamp's Ashburn and Chicago data centers dropped for short periods; monitoring showed the newer link down for about 20-30 seconds.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingIn-memory cacheDatabase replication

   Contributing factor

   Single point of failureRetry stormSignal buried in noise

   Detection

   A flood of alerts from PagerDuty, Nagios and Prometheus, with monitoring showing the lost metro link. A lingering alert for one Redis DB instance later pointed to the replication problem.

   Remediation

   • Turned chat off at the load balancer layer so the rest of BC3 could stabilise
   • Restarted Redis on the replica that was stuck in the resynchronization loop

   Action items

   • Make the documented Redis configuration changes for the known replication issue
   • Make concrete plans to remove the shared physical path under the two metro links
   • Prove the failover systems in production at peak load
   • Make explicit room for post-incident follow-up work and push other work out

   Lessons

   Planned redundancy and live testing had given Basecamp a false sense of security, especially at peak load: moving between redundant network links or shifting load across data centers should work without issue, and Basecamp would rather find out in a controlled test with the full team ready. The follow-up from an outage needs urgency and explicitly cleared time, not just a place on a list.

   View case
2. IL-0215 Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedly

   Twilio2013

   Symptoms

   At 2:39 AM PDT (9:39 UTC) on 18 July 2013, services that relied on the redis-master of Twilio's billing system began to fail because the load on its host had become extreme. At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster.

   Database overloadData lossWrites rejected

   Trigger

   At 1:35 AM PDT on 18 July a loss of network connectivity disconnected all billing redis-slaves from the redis-master at once, and all of them requested a full synchronization at the same time. On-call engineers then restarted the overloaded redis-master.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Retry stormHarmful manual actionUnsafe failure behaviour

   Detection

   At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster. At 3:28 AM PDT the billing system monitoring reported the anomalous charges, and at 12:36 PM PDT monitoring detected the recurrence.

   Remediation

   • Restarted the redis-master to relieve the high load, which loaded a wrong configuration
   • Took the billing system offline to stop further credit card charges
   • Restored service to all suspended accounts
   • Restored the lost account balances from the independent double-bookkeeping record in the relational datastore
   • Took billing offline again after the recurrence and reactivated the newly suspended accounts
   • Refunded all erroneous credit card charges
   • Gave all affected accounts a credit of 10% of their last 30 days of spend
   • Replaced the original Redis cluster and corrected the redis-master configuration
   • Activated the billing system progressively for groups of customer accounts

   Action items

   • Disable Redis restarts on the redis-master and recover the master by pivoting a slave instead
   • Add fail-safes so the billing system neither suspends accounts nor charges cards when balances are missing or cannot be written
   • Validate the billing system against the double-bookkeeping databases in real time

   Lessons

   Losing the in-flight balances and the ability to write balances at the same time exposed a flaw in the auto-recharge system: it failed in a dangerous way, charging cards and suspending accounts instead of stopping.

   View case

By root cause2