Cases3 cases

1 to 3 of 3

1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes

   GitLab2023

   Symptoms

   GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Failover failureHigh error rate

   Trigger

   The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

   Root cause

   Cloud provider outage

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Remediation

   • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary

   Action items

   • Alert when the Sentinel service is down
   • Make merge requests robust to Redis Sidekiq outages

   Lessons

   An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

   View case
2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper

   AppOptics2018

   Symptoms

   On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

   High error rateFailover failureMonitoring outage

   Trigger

   A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

   Root cause

   Cloud provider outage

   Component

   Message brokerCoordination and key-value store

   Contributing factor

   Single point of failureExcessive blast radiusMonitoring blind spot

   Remediation

   • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
   • Restarted every Kafka broker in a rolling restart until all partitions were back online
   • Worked through the backlog, including the backlog of alerts, and resumed normal operations

   Action items

   • Isolate the Kafka clusters better across availability zones and regions
   • Stop sharing one ZooKeeper ensemble across Kafka clusters
   • Run game days on this failure and check whether newer Kafka versions fix it
   • Put in place more checks that notify the team earlier about subsystems that need extra help to recover

   Lessons

   Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

   View case
3. IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutes

   Stack Exchange2017

   Symptoms

   On January 24, 2017 at 17:53 UTC the Stack Exchange network, including Stack Overflow, went into read-only mode for about 5 minutes and then went offline for 12 minutes. It took the team 2 minutes to notice the problem, 5 minutes to locate its source and 10 minutes to restore service.

   Writes rejectedService outageFailover failureData loss

   Trigger

   At 17:53 UTC the primary SQL Server, NY-SQL02, triggered a bugcheck in the SQL Server process.

   Root cause

   unknown

   Component

   Database

   Contributing factor

   Single point of failure

   Remediation

   • Restarted the SQL service on NY-SQL02, which brought the network back online in read-only mode
   • Put the sites back into read-write mode after a sanity check of SQL health
   • Brought the network online without waiting for the memory dump to complete
   • Took NY-SQL02 out of production for testing, including a memory test, and ran the normal load on NY-SQL01
   • Brought the SQL cluster up to 2016 SP1 CU1

   View case

By root cause2