Incident Library
Applied filters ComponentIn-memory cacheFactorSingle point of failureOrganisationGitLabClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes
    GitLab2023
    Symptoms

    GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

    Failover failureHigh error rate
    Trigger

    The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

    Root cause
    Cloud provider outage
    Component
    In-memory cacheDatabase failover manager
    Contributing factor
    Single point of failureMonitoring blind spot
    Detection

    Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

    Remediation
    • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary
    Action items
    • Alert when the Sentinel service is down
    • Make merge requests robust to Redis Sidekiq outages
    Lessons

    An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

    View case
Check your cluster for this

This case involves In-memory cache and Database failover manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
In-memory cacheDatabase failover manager
Contributing factor
Monitoring blind spotSingle point of failure
Organisation
GitLab
  1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes
    GitLab2023