Incident Library
Applied filters Root causeCloud provider outageSymptomHigh error rateFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes
    GitLab2023
    Symptoms

    GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

    Failover failureHigh error rate
    Trigger

    The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

    Root cause
    Cloud provider outage
    Component
    In-memory cacheDatabase failover manager
    Contributing factor
    Single point of failureMonitoring blind spot
    Detection

    Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

    Remediation
    • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary
    Action items
    • Alert when the Sentinel service is down
    • Make merge requests robust to Redis Sidekiq outages
    Lessons

    An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

    View case
  2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018
    Symptoms

    On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

    High error rateFailover failureMonitoring outage
    Trigger

    A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

    Root cause
    Cloud provider outage
    Component
    Message brokerCoordination and key-value store
    Contributing factor
    Single point of failureExcessive blast radiusMonitoring blind spot
    Remediation
    • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
    • Restarted every Kafka broker in a rolling restart until all partitions were back online
    • Worked through the backlog, including the backlog of alerts, and resumed normal operations
    Action items
    • Isolate the Kafka clusters better across availability zones and regions
    • Stop sharing one ZooKeeper ensemble across Kafka clusters
    • Run game days on this failure and check whether newer Kafka versions fix it
    • Put in place more checks that notify the team earlier about subsystems that need extra help to recover
    Lessons

    Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

    View case
  3. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded
    PagerDuty2013sev1
    Symptoms

    On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

    High latencyHigh error rateIntermittent connection failures
    Trigger

    At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

    Root cause
    Cloud provider outage
    Component
    Transit gateway or network interconnectCoordination and key-value store
    Contributing factor
    Single point of failureUnverified assumptionMonitoring blind spot
    Detection

    The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

    Remediation
    • Restarted most of the failed coordinator processes by hand
    • The notification dispatch system reached quorum again and processed all queued notifications
    Action items
    • Add more logging and aggregate logs into a single searchable source
    • Add a process watcher that restarts failed coordinator processes automatically
    • Build a dashboard that shows inter-host connectivity
    • Train engineering staff on Cassandra and ZooKeeper
    • Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure
    Lessons

    Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

    View case
Check your cluster for this

These 3 cases involve Coordination and key-value store and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage3 cases

Component
Coordination and key-value store2In-memory cache1Database failover manager1Transit gateway or network interconnect1Message broker1
Contributing factor
Monitoring blind spot3Single point of failure3Unverified assumption1Excessive blast radius1
Organisation
AppOptics1GitLab1PagerDuty1
  1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes
    GitLab2023
  2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018
  3. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded
    PagerDuty2013