Incident Library

Organisation

PagerDuty

PagerDuty · pagerduty

3 cases · 2013 to 2025

Root cause Capacity exhaustion1Cloud provider outage1Faulty deployment1
Component Coordination and key-value store1Database1Transit gateway or network interconnect1Message broker1
Contributing factor Excessive blast radius2Monitoring blind spot2Signal buried in noise1Unverified assumption1Change with a delayed effect1Ignored early signal1Harmful manual action1No headroom1Single point of failure1
Symptom High error rate3High latency2Database overload1Intermittent connection failures1Memory pressure1Throughput degradation1

Cases

  1. IL-0214 PagerDuty's Kafka brokers ran out of JVM heap one after another, and the event APIs rejected events with 502 responsesPagerDuty · 2025

    Root cause Faulty deployment

  2. IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming eventsPagerDuty · 2014 · sev1

    Root cause Capacity exhaustion

  3. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degradedPagerDuty · 2013 · sev1

    Root cause Cloud provider outage