Cases13 cases

10 to 13 of 13

1. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper

   AppOptics2018

   Symptoms

   On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

   High error rateFailover failureMonitoring outage

   Trigger

   A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

   Root cause

   Cloud provider outage

   Component

   Message brokerCoordination and key-value store

   Contributing factor

   Single point of failureExcessive blast radiusMonitoring blind spot

   Remediation

   • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
   • Restarted every Kafka broker in a rolling restart until all partitions were back online
   • Worked through the backlog, including the backlog of alerts, and resumed normal operations

   Action items

   • Isolate the Kafka clusters better across availability zones and regions
   • Stop sharing one ZooKeeper ensemble across Kafka clusters
   • Run game days on this failure and check whether newer Kafka versions fix it
   • Put in place more checks that notify the team earlier about subsystems that need extra help to recover

   Lessons

   Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

   View case
2. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours

   Parse.ly2016

   Symptoms

   On Thursday 9 June 2016 at 11:30am ET, real-time data suddenly stopped arriving in Parse.ly's dashboard. Only the data delivery monitors reported a failure; no hardware or machine had failed and no other system reported anything, so the cause was at first unclear.

   Service outage

   Trigger

   A regular Chef run that applied a configuration change restarted the service on two of the three nodes of a ZooKeeper cluster at the same moment.

   Root cause

   Faulty deployment

   Component

   Coordination and key-value store

   Contributing factor

   Unverified assumptionNo staged rollout

   Detection

   Parse.ly's data delivery monitors reported that data had stopped flowing to the dashboard; no other system reported a failure.

   Remediation

   • Cleared a few hours of data from the live systems and cold-started the clusters, taking them offline and bringing them back from scratch
   • Delivered the data from the outage window late through the historical rebuild process

   Action items

   • Separate production configuration changes from production node restarts
   • Always keep a plan B for rapid recovery in systems and processes, instead of treating multi-node failures as impossible

   Lessons

   Automatic routine service restarts are dangerous for services that form a cluster across nodes, and jittering configuration runs only makes simultaneous restarts unlikely, not impossible, once a fleet has hundreds of nodes. A system designed to survive single-node failures still needs a recovery plan for multi-node failures.

   View case
3. IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster

   Amazon Web Services2014

   Symptoms

   From 9:16 PM to 11:16 PM PDT on June 13, Amazon SimpleDB in the US East Region was unavailable to all API calls except a fraction of eventually consistent reads, and SimpleDB API requests returned server-side errors (http response code 500).

   Service outageHigh error rate

   Trigger

   Power was lost to the servers that hosted several SimpleDB storage nodes in a single data center, and those storage nodes became unavailable at the same time.

   Root cause

   Software defect

   Component

   DatabaseCoordination and key-value storeData center power and cooling

   Contributing factor

   Automation amplification

   Remediation

   • Raised the lock service handshake timeout values by hand
   • Restarted a subset of metadata nodes, which could then authorize the storage nodes to rejoin the cluster
   • Throttled CreateDomain and DeleteDomain calls until 1:30 AM while the remaining metadata nodes recovered

   Action items

   • Set a longer lock service handshake timeout in all SimpleDB Regions
   • Make nodes wait and retry handshakes later with a longer timeout instead of removing themselves from the cluster after repeated timeouts

   Lessons

   The short handshake timeout had been chosen to detect replica failures quickly, and AWS concluded it was too low. Nodes removing themselves from the cluster right after several handshake timeouts widened the event and caused the API errors; the nodes should have waited and retried with a longer timeout.

   View case
4. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded

   PagerDuty2013sev1

   Symptoms

   On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

   High latencyHigh error rateIntermittent connection failures

   Trigger

   At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

   Root cause

   Cloud provider outage

   Component

   Transit gateway or network interconnectCoordination and key-value store

   Contributing factor

   Single point of failureUnverified assumptionMonitoring blind spot

   Detection

   The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

   Remediation

   • Restarted most of the failed coordinator processes by hand
   • The notification dispatch system reached quorum again and processed all queued notifications

   Action items

   • Add more logging and aggregate logs into a single searchable source
   • Add a process watcher that restarts failed coordinator processes automatically
   • Build a dashboard that shows inter-host connectivity
   • Train engineering staff on Cassandra and ZooKeeper
   • Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure

   Lessons

   Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

   View case

By root cause6