Cases2 cases

1 to 2 of 2

1. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours

   Parse.ly2016

   Symptoms

   On Thursday 9 June 2016 at 11:30am ET, real-time data suddenly stopped arriving in Parse.ly's dashboard. Only the data delivery monitors reported a failure; no hardware or machine had failed and no other system reported anything, so the cause was at first unclear.

   Service outage

   Trigger

   A regular Chef run that applied a configuration change restarted the service on two of the three nodes of a ZooKeeper cluster at the same moment.

   Root cause

   Faulty deployment

   Component

   Coordination and key-value store

   Contributing factor

   Unverified assumptionNo staged rollout

   Detection

   Parse.ly's data delivery monitors reported that data had stopped flowing to the dashboard; no other system reported a failure.

   Remediation

   • Cleared a few hours of data from the live systems and cold-started the clusters, taking them offline and bringing them back from scratch
   • Delivered the data from the outage window late through the historical rebuild process

   Action items

   • Separate production configuration changes from production node restarts
   • Always keep a plan B for rapid recovery in systems and processes, instead of treating multi-node failures as impossible

   Lessons

   Automatic routine service restarts are dangerous for services that form a cluster across nodes, and jittering configuration runs only makes simultaneous restarts unlikely, not impossible, once a fleet has hundreds of nodes. A system designed to survive single-node failures still needs a recovery plan for multi-node failures.

   View case
2. IL-0275 Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hours

   Parse.ly2015

   Symptoms

   On Thursday night, 26 March 2015, and again on Friday, 27 March, Parse.ly's data processing backend stopped. Monitoring tools such as Munin, Sentry and Pingdom reported failures widely, PagerDuty woke the on-call engineer, and no new analytics data appeared in the dashboards at dash.parsely.com and preview.parsely.com for several hours.

   Service outageDNS resolution failures

   Trigger

   Data volume reached a point where the Kafka brokers, running on modest Amazon EC2 nodes without high network capacity, hit operating system network limits under the outbound read traffic and became unavailable.

   Root cause

   Capacity exhaustion

   Component

   Message brokerService discovery

   Contributing factor

   Monitoring blind spotUnknown limitNo headroomHarmful manual action

   Detection

   Munin, Sentry and Pingdom reported the failures widely, Graphite graphs showed them, and PagerDuty woke the on-call engineer when the first outages started in the night.

   Remediation

   • Added new brokers to expand the Kafka cluster
   • Corrected the configuration of the new brokers and deployed them to the cluster, after which network utilization per node dropped and processing resumed
   • Turned the downstream consumer systems back on slowly through Apache Storm and reprocessed the backlog
   • Began restoring the data to customer accounts from the raw data archived in Amazon S3

   Lessons

   CPU and disk I/O are good signs of machine health, but creeping network utilization is harder to monitor, because 100% of network capacity is not well defined on cloud VMs that share host machines. When every node of a cluster runs close to the same limit, losing one node can take down the whole cluster, so failure planning should cover an entire subsystem going offline, not just one or two machines. Keeping raw data archival in simple files and S3 uploads, separate from the Kafka stream, meant there was no risk of permanent data loss.

   View case

By root cause2