Cases2 cases

1 to 2 of 2

1. IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events

   PagerDuty2014sev1

   Symptoms

   On 3 June 2014, PagerDuty's notification pipeline performed poorly and some notifications were delayed after the Cassandra cluster behind it became heavily degraded. About 3% of events sent to the integration API could not be received.

   Database overloadHigh error rateHigh latency

   Trigger

   On 3 June 2014 the Cassandra background repair process started its normal run on one node of the cluster while other high load was on the cluster; on 4 June the team restarted the postponed repair by hand.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomExcessive blast radiusHarmful manual actionIgnored early signal

   Remediation

   • Reduced the load on the Cassandra cluster
   • Stopped the Cassandra repair process
   • Cut communication between some Cassandra nodes to stabilise the cluster
   • On 4 June, deleted all in-flight data in the notification pipeline in a "factory reset"
   • Brought downstream systems back into a consistent state with the emptied pipeline by hand

   Action items

   • Scale the existing Cassandra nodes vertically to bigger, faster servers
   • Set up multiple Cassandra clusters and spread the workloads across them
   • Set system load thresholds at which the Cassandra clusters are scaled out horizontally ahead of need
   • Upgrade the current and new clusters to a more recent Cassandra version
   • Add more load shedding techniques to control Cassandra under high load
   • Bring more Cassandra expertise in-house

   Lessons

   PagerDuty concluded that its Cassandra cluster was underscaled and shared by too many services with different load and usage patterns, and that the scaling work should have come before the planned Cassandra version upgrade.

   View case
2. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded

   PagerDuty2013sev1

   Symptoms

   On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

   High latencyHigh error rateIntermittent connection failures

   Trigger

   At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

   Root cause

   Cloud provider outage

   Component

   Transit gateway or network interconnectCoordination and key-value store

   Contributing factor

   Single point of failureUnverified assumptionMonitoring blind spot

   Detection

   The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

   Remediation

   • Restarted most of the failed coordinator processes by hand
   • The notification dispatch system reached quorum again and processed all queued notifications

   Action items

   • Add more logging and aggregate logs into a single searchable source
   • Add a process watcher that restarts failed coordinator processes automatically
   • Build a dashboard that shows inter-host connectivity
   • Train engineering staff on Cassandra and ZooKeeper
   • Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure

   Lessons

   Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

   View case

By root cause2