Cases2 cases

1 to 2 of 2

1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes

   GitLab2023

   Symptoms

   GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Failover failureHigh error rate

   Trigger

   The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

   Root cause

   Cloud provider outage

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Remediation

   • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary

   Action items

   • Alert when the Sentinel service is down
   • Make merge requests robust to Redis Sidekiq outages

   Lessons

   An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

   View case
2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper

   AppOptics2018

   Symptoms

   On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

   High error rateFailover failureMonitoring outage

   Trigger

   A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

   Root cause

   Cloud provider outage

   Component

   Message brokerCoordination and key-value store

   Contributing factor

   Single point of failureExcessive blast radiusMonitoring blind spot

   Remediation

   • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
   • Restarted every Kafka broker in a rolling restart until all partitions were back online
   • Worked through the backlog, including the backlog of alerts, and resumed normal operations

   Action items

   • Isolate the Kafka clusters better across availability zones and regions
   • Stop sharing one ZooKeeper ensemble across Kafka clusters
   • Run game days on this failure and check whether newer Kafka versions fix it
   • Put in place more checks that notify the team earlier about subsystems that need extra help to recover

   Lessons

   Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

   View case

By root cause1