Cases6 cases

5 to 6 of 6

1. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down

   Instapaper2017

   Symptoms

   On Wednesday, February 9, 2017 at 12:30PM PT, inserts of new entries into the bookmarks table of Instapaper's MySQL database on Amazon RDS started to fail with an error. The bookmarks table, which stores the articles Instapaper users save, had grown past 2TB, and the Instapaper service was down until Thursday, February 10 at 7:30pm PT.

   Writes rejectedService outage

   Trigger

   At 12:30PM PT on February 9 the bookmarks table grew past the 2TB file size limit of the instance's filesystem.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase backups and snapshotsDatabase replication

   Contributing factor

   Unknown limitMonitoring blind spotUntested backupNo runbookLimited debug access

   Remediation

   • Dumped the production database to a 5.5TB RAID-0 disk on an i2.8xlarge instance with Pinterest's SRE team
   • Launched a temporary production instance with limited access to archives
   • Set up an Aurora read replica of the old database as one recovery path
   • Imported the data into a new MySQL RDS instance without secondary indexes, then built the indexes
   • Had an Amazon engineer rsync the data from the ext3 filesystem to an ext4 filesystem mounted on the failed database
   • Synced the new ext4 database with the temporary production database through row-based replication
   • Promoted the new database to master and pointed the application at it

   Action items

   • Define a workflow that escalates system-wide Instapaper outages immediately to Pinterest's SRE team
   • Test MySQL backups every month instead of every three months
   • Put a process in place with Pinterest's SRE team to recover from this type of failure

   Lessons

   A filesystem limit that Instapaper did not know about and could not see made both the production database and all its backups useless, and the only way out was to restore the data onto a new filesystem. Because the hosted database could only be reached through MySQL, filesystem-level fixes such as rsync needed Amazon engineers. Even with a perfect response, rebuilding the database would have taken at least 10 hours. The lack of a disaster recovery plan for this kind of failure, and the estimate of the dump time from row counts, made the downtime and recovery longer than necessary, and the response could have drawn better on Pinterest's and AWS's people. The new escalation workflow and monthly backup tests would not have prevented this failure but would speed up the response to a future one.

   View case
2. IL-0296 Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customers

   Transloadit2016

   Symptoms

   Reports came in of more WORKER_JOB_ERROR errors, which tell Transloadit there is a communication problem between its Uploaders, the machines that receive files and orchestrate Assemblies, and its Drones, the machines that run encoding tasks. Many Assemblies became much slower because retries with exponential back-off kicked in, and a few could not be replayed.

   High error rateHigh latencyLoad imbalance

   Trigger

   A steep increase in HLS traffic, together with more Drones online processing the same volume faster, which put more load on the central Redis nodes.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Monitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Reports of an increased number of WORKER_JOB_ERROR errors.

   Remediation

   • Re-added lost messages and relaxed heartbeat monitoring as a stopgap
   • Upgraded Redis to the latest stable version
   • Doubled the Redis cluster to six nodes and then added three more, which made it worse
   • Brought the number of read replicas down to one, which cut traffic and failure rates
   • Capped how far Drones scale up in each region

   Action items

   • Add logging and alarms for approaching saturation
   • Find ways to decentralise the traffic between the Redis nodes and the Drones and Uploaders
   • Add sharding per Uploader on top of the sharding per region

   Lessons

   To handle more traffic Transloadit had to use fewer Redis nodes, not more: each added replica multiplied the replication traffic from the single leader that takes every write, so scaling up the number of nodes was the wrong call. Bandwidth had not been among the monitored Redis metrics.

   View case

By root cause1