Incident Library
Applied filters Root causeCloud provider outageSymptomFailover failureFactorSingle point of failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes
    GitLab2023
    Symptoms

    GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

    Failover failureHigh error rate
    Trigger

    The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

    Root cause
    Cloud provider outage
    Component
    In-memory cacheDatabase failover manager
    Contributing factor
    Single point of failureMonitoring blind spot
    Detection

    Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

    Remediation
    • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary
    Action items
    • Alert when the Sentinel service is down
    • Make merge requests robust to Redis Sidekiq outages
    Lessons

    An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

    View case
  2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018
    Symptoms

    On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

    High error rateFailover failureMonitoring outage
    Trigger

    A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

    Root cause
    Cloud provider outage
    Component
    Message brokerCoordination and key-value store
    Contributing factor
    Single point of failureExcessive blast radiusMonitoring blind spot
    Remediation
    • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
    • Restarted every Kafka broker in a rolling restart until all partitions were back online
    • Worked through the backlog, including the backlog of alerts, and resumed normal operations
    Action items
    • Isolate the Kafka clusters better across availability zones and regions
    • Stop sharing one ZooKeeper ensemble across Kafka clusters
    • Run game days on this failure and check whether newer Kafka versions fix it
    • Put in place more checks that notify the team earlier about subsystems that need extra help to recover
    Lessons

    Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

    View case
Check your cluster for this

These 2 cases involve In-memory cache and Coordination and key-value store. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage2 cases

Component
In-memory cache1Coordination and key-value store1Database failover manager1Message broker1
Contributing factor
Monitoring blind spot2Single point of failure2Excessive blast radius1
Organisation
AppOptics1GitLab1
  1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes
    GitLab2023
  2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018