Cases9 cases

8 to 9 of 9

1. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours

   Parse.ly2016

   Symptoms

   On Thursday 9 June 2016 at 11:30am ET, real-time data suddenly stopped arriving in Parse.ly's dashboard. Only the data delivery monitors reported a failure; no hardware or machine had failed and no other system reported anything, so the cause was at first unclear.

   Service outage

   Trigger

   A regular Chef run that applied a configuration change restarted the service on two of the three nodes of a ZooKeeper cluster at the same moment.

   Root cause

   Faulty deployment

   Component

   Coordination and key-value store

   Contributing factor

   Unverified assumptionNo staged rollout

   Detection

   Parse.ly's data delivery monitors reported that data had stopped flowing to the dashboard; no other system reported a failure.

   Remediation

   • Cleared a few hours of data from the live systems and cold-started the clusters, taking them offline and bringing them back from scratch
   • Delivered the data from the outage window late through the historical rebuild process

   Action items

   • Separate production configuration changes from production node restarts
   • Always keep a plan B for rapid recovery in systems and processes, instead of treating multi-node failures as impossible

   Lessons

   Automatic routine service restarts are dangerous for services that form a cluster across nodes, and jittering configuration runs only makes simultaneous restarts unlikely, not impossible, once a fleet has hundreds of nodes. A system designed to survive single-node failures still needs a recovery plan for multi-node failures.

   View case
2. IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster

   Amazon Web Services2014

   Symptoms

   From 9:16 PM to 11:16 PM PDT on June 13, Amazon SimpleDB in the US East Region was unavailable to all API calls except a fraction of eventually consistent reads, and SimpleDB API requests returned server-side errors (http response code 500).

   Service outageHigh error rate

   Trigger

   Power was lost to the servers that hosted several SimpleDB storage nodes in a single data center, and those storage nodes became unavailable at the same time.

   Root cause

   Software defect

   Component

   DatabaseCoordination and key-value storeData center power and cooling

   Contributing factor

   Automation amplification

   Remediation

   • Raised the lock service handshake timeout values by hand
   • Restarted a subset of metadata nodes, which could then authorize the storage nodes to rejoin the cluster
   • Throttled CreateDomain and DeleteDomain calls until 1:30 AM while the remaining metadata nodes recovered

   Action items

   • Set a longer lock service handshake timeout in all SimpleDB Regions
   • Make nodes wait and retry handshakes later with a longer timeout instead of removing themselves from the cluster after repeated timeouts

   Lessons

   The short handshake timeout had been chosen to detect replica failures quickly, and AWS concluded it was too low. Nodes removing themselves from the cluster right after several handshake timeouts widened the event and caused the API errors; the nodes should have waited and retried with a longer timeout.

   View case

By root cause4