Incident Library
Applied filters Root causeCloud provider outageComponentCoordination and key-value storeFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018
    Symptoms

    On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

    High error rateFailover failureMonitoring outage
    Trigger

    A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

    Root cause
    Cloud provider outage
    Component
    Message brokerCoordination and key-value store
    Contributing factor
    Single point of failureExcessive blast radiusMonitoring blind spot
    Remediation
    • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
    • Restarted every Kafka broker in a rolling restart until all partitions were back online
    • Worked through the backlog, including the backlog of alerts, and resumed normal operations
    Action items
    • Isolate the Kafka clusters better across availability zones and regions
    • Stop sharing one ZooKeeper ensemble across Kafka clusters
    • Run game days on this failure and check whether newer Kafka versions fix it
    • Put in place more checks that notify the team earlier about subsystems that need extra help to recover
    Lessons

    Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

    View case
  2. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded
    PagerDuty2013sev1
    Symptoms

    On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

    High latencyHigh error rateIntermittent connection failures
    Trigger

    At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

    Root cause
    Cloud provider outage
    Component
    Transit gateway or network interconnectCoordination and key-value store
    Contributing factor
    Single point of failureUnverified assumptionMonitoring blind spot
    Detection

    The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

    Remediation
    • Restarted most of the failed coordinator processes by hand
    • The notification dispatch system reached quorum again and processed all queued notifications
    Action items
    • Add more logging and aggregate logs into a single searchable source
    • Add a process watcher that restarts failed coordinator processes automatically
    • Build a dashboard that shows inter-host connectivity
    • Train engineering staff on Cassandra and ZooKeeper
    • Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure
    Lessons

    Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

    View case
Check your cluster for this

These 2 cases involve Coordination and key-value store and Transit gateway or network interconnect. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage2 cases

Component
Coordination and key-value store2Transit gateway or network interconnect1Message broker1
Contributing factor
Monitoring blind spot2Single point of failure2Unverified assumption1Excessive blast radius1
Organisation
AppOptics1PagerDuty1
  1. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018
  2. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded
    PagerDuty2013