Cases2 cases

1 to 2 of 2

1. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read

   Google Cloud2017

   Symptoms

   From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

   Service outageDatabase overloadHigh latencyHigh error rate

   Trigger

   A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase

   Contributing factor

   Ineffective rollbackExcessive blast radius

   Detection

   Google received an automated alert at 12:34 US/Pacific, one minute after the start.

   Remediation

   • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
   • Ramped Memcache traffic back up gradually while the error rate fell

   Action items

   • Reduce the number of readers of the global configuration to avoid contention during writes
   • Regionalise the configuration to limit the blast radius of similar failures

   View case
2. IL-0215 Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedly

   Twilio2013

   Symptoms

   At 2:39 AM PDT (9:39 UTC) on 18 July 2013, services that relied on the redis-master of Twilio's billing system began to fail because the load on its host had become extreme. At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster.

   Database overloadData lossWrites rejected

   Trigger

   At 1:35 AM PDT on 18 July a loss of network connectivity disconnected all billing redis-slaves from the redis-master at once, and all of them requested a full synchronization at the same time. On-call engineers then restarted the overloaded redis-master.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Retry stormHarmful manual actionUnsafe failure behaviour

   Detection

   At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster. At 3:28 AM PDT the billing system monitoring reported the anomalous charges, and at 12:36 PM PDT monitoring detected the recurrence.

   Remediation

   • Restarted the redis-master to relieve the high load, which loaded a wrong configuration
   • Took the billing system offline to stop further credit card charges
   • Restored service to all suspended accounts
   • Restored the lost account balances from the independent double-bookkeeping record in the relational datastore
   • Took billing offline again after the recurrence and reactivated the newly suspended accounts
   • Refunded all erroneous credit card charges
   • Gave all affected accounts a credit of 10% of their last 30 days of spend
   • Replaced the original Redis cluster and corrected the redis-master configuration
   • Activated the billing system progressively for groups of customer accounts

   Action items

   • Disable Redis restarts on the redis-master and recover the master by pivoting a slave instead
   • Add fail-safes so the billing system neither suspends accounts nor charges cards when balances are missing or cannot be written
   • Validate the billing system against the double-bookkeeping databases in real time

   Lessons

   Losing the in-flight balances and the ability to write balances at the same time exposed a flaw in the auto-recharge system: it failed in a dangerous way, charging cards and suspending accounts instead of stopping.

   View case

By root cause1