Cases2 cases

1 to 2 of 2

1. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours

   Parse.ly2016

   Symptoms

   On Thursday 9 June 2016 at 11:30am ET, real-time data suddenly stopped arriving in Parse.ly's dashboard. Only the data delivery monitors reported a failure; no hardware or machine had failed and no other system reported anything, so the cause was at first unclear.

   Service outage

   Trigger

   A regular Chef run that applied a configuration change restarted the service on two of the three nodes of a ZooKeeper cluster at the same moment.

   Root cause

   Faulty deployment

   Component

   Coordination and key-value store

   Contributing factor

   Unverified assumptionNo staged rollout

   Detection

   Parse.ly's data delivery monitors reported that data had stopped flowing to the dashboard; no other system reported a failure.

   Remediation

   • Cleared a few hours of data from the live systems and cold-started the clusters, taking them offline and bringing them back from scratch
   • Delivered the data from the outage window late through the historical rebuild process

   Action items

   • Separate production configuration changes from production node restarts
   • Always keep a plan B for rapid recovery in systems and processes, instead of treating multi-node failures as impossible

   Lessons

   Automatic routine service restarts are dangerous for services that form a cluster across nodes, and jittering configuration runs only makes simultaneous restarts unlikely, not impossible, once a fleet has hundreds of nodes. A system designed to survive single-node failures still needs a recovery plan for multi-node failures.

   View case
2. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded

   PagerDuty2013sev1

   Symptoms

   On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

   High latencyHigh error rateIntermittent connection failures

   Trigger

   At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

   Root cause

   Cloud provider outage

   Component

   Transit gateway or network interconnectCoordination and key-value store

   Contributing factor

   Single point of failureUnverified assumptionMonitoring blind spot

   Detection

   The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

   Remediation

   • Restarted most of the failed coordinator processes by hand
   • The notification dispatch system reached quorum again and processed all queued notifications

   Action items

   • Add more logging and aggregate logs into a single searchable source
   • Add a process watcher that restarts failed coordinator processes automatically
   • Build a dashboard that shows inter-host connectivity
   • Train engineering staff on Cassandra and ZooKeeper
   • Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure

   Lessons

   Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

   View case

By root cause2