IL-0362 PagerDuty · 2013-04-13awsnetworking

PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded

PagerDuty ran in three datacenters, two AWS regions and one at Linode, and had spent a year rearchitecting its software to survive the loss of a whole datacenter, including a network partition, but not of two at once. The three were chosen to be physically separate with no dependency between them, yet two of them turned out to share a common peering point. On 13 April 2013 at 7:57am Pacific Time that peering point in Northern California began to degrade, and both AWS datacenters went offline, leaving PagerDuty with only the Linode datacenter.

From 8:11 the on-call engineer was paged about a node of the notification dispatch system, and by 8:18 notifications showed higher latency and error rates. At 8:35 high network latency stopped the dispatch system from reaching quorum and no notifications went out until 8:53, when quorum returned and queued notifications were processed; the events API kept accepting events. Most failed coordinator processes were restarted by hand, and the post-mortem found that a misconfiguration of the coordinator service slowed recovery. AWS reported the peering point connectivity issue ended at 9:23. The analysis also found that some systems lacked the logging needed to debug them, that there was no good view of inter-host connectivity, and that not all engineers were current on Cassandra and ZooKeeper.

Severity: sev1
Scope: Notifications were delayed and then not dispatched for 18 minutes between 8:35am and 8:53am Pacific Time; API endpoints returned 500 errors; the events API kept accepting events
Users affected: –
Region: two AWS regions sharing a peering point in Northern California
Time to detect: 14m
Time to resolve: 56m

Symptoms

On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

At 8:18 PagerDuty's monitoring detected failures across multiple notification providers, caused by the connectivity problem; most notifications still went through but with higher latency and error rates. A Sev-2 was declared at 8:31 and more engineers were paged. At 8:35 the notification dispatch system lost the ability to dispatch at all because it could not establish quorum under the high network latency, and a Sev-1 was declared. The events API kept accepting events throughout.

At 8:53 the dispatch system reached quorum again and began processing the queued notifications, 18 minutes after dispatch stopped. Most of the coordinator processes that had failed were restarted by hand, and a misconfiguration of the coordinator service kept PagerDuty from recovering faster.

Symptom High latency High error rate Intermittent connection failures

Root cause

Root cause Cloud provider outage

Component Transit gateway or network interconnect Coordination and key-value store

Contributing factor Single point of failure Unverified assumption Monitoring blind spot

Trigger: At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

Detection

The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

Remediation

Restarted most of the failed coordinator processes by hand
The notification dispatch system reached quorum again and processed all queued notifications

Action items

Add more logging and aggregate logs into a single searchable source
Add a process watcher that restarts failed coordinator processes automatically
Build a dashboard that shows inter-host connectivity
Train engineering staff on Cassandra and ZooKeeper
Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure

Lessons

Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

Sources

https://www.pagerduty.com/blog/outage-post-mortem-april-13-2013/ · blog · archived copy

Related cases

IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were downPython Software Foundation · 2022
same root cause · shared Coordination and key-value store
IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went downSlack · 2021
same root cause · shared Transit gateway or network interconnect
IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeperAppOptics · 2018
same root cause · shared Coordination and key-value store

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Transit gateway or network interconnect, Coordination and key-value store among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0362
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0362/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.