Incident Library

IL-0291 GitLab · 2023-01-05 (published)databasegcp

GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes

In GitLab.com's redis-sidekiq Redis, managed by Redis Sentinel, the node redis-sidekiq-01-db-gprd had been down for more than 30 days, so the cluster had been running with a quorum of 2. An alert that would have shown a down node had been added in the runbooks and later removed for being too noisy. When the primary, redis-sidekiq-03-db-gprd, failed on a GCP host error and its VM was reset and moved to another node, no quorum was possible. The services on the nodes do not start automatically on boot, so the cluster was left with one replica stuck on a quorum conflict.

Once responders saw there were not enough machines to elect a primary, they restarted the failed node. Redis was out for about 30 minutes. The best estimate of affected customers is 16,621, and the review says the real number may have been much larger. About 4,000 merge requests stayed unusable for several hours, which made up the long tail of the recovery.

Severity
–
Scope
All services using Sidekiq jobs on GitLab.com fully down for 30 minutes, with knock-on errors in Web and API requests and SAML login; about 4,000 merge requests unusable for several hours
Users affected
External customers; best estimate 16,621, possibly much more
Region
–
Time to detect
–
Time to resolve
–

Symptoms

GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

Every service that uses Sidekiq jobs was fully down for those 30 minutes, with knock-on effects on Web and API requests; among other things, users could not log in with SAML. For about 4,000 merge requests created during the outage, diffs, pipeline status and merging stayed unavailable for several hours afterwards, because merge requests could not recover by themselves from a Redis Sidekiq outage.

Symptom Failover failureHigh error rate

Root cause

Root cause Cloud provider outage

Component In-memory cacheDatabase failover manager
Contributing factor Single point of failureMonitoring blind spot
Trigger
The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

Detection

Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

Remediation

Action items

Lessons

An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

Sources

References

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, In-memory cache, Database failover manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.