Cases2 cases

1 to 2 of 2

1. IL-0290 GitLab.com's Redis Sidekiq cluster ran out of memory and lost quorum, causing 500 errors and a 12 million job backlog

   GitLab2026sev1

   Symptoms

   From 17:12 UTC on 12 May 2026, error rates rose at the same time across GitLab.com's web, Git, API and CI runner services. Users got frequent 500 errors, failed or stuck CI pipelines, authentication failures and unreliable Git access, and a Prometheus Alertmanager alert opened the incident at 17:15.

   High error rateMemory pressureThroughput degradation

   Trigger

   Earlier on 12 May, as a mitigation for a previous incident, INC-10084, a Sidekiq queueing apdex SLO violation on shard catchall-b, jobs of AuditEvents::AuditEventStreamingWorker were deferred. The deferred audit event jobs then piled up in the Redis Sidekiq cluster.

   Root cause

   Capacity exhaustion

   Component

   In-memory cache

   Contributing factor

   Change with a delayed effectPages not deliveredLimited debug access

   Detection

   A Prometheus Alertmanager alert reported the incident at 17:15:10 UTC, three minutes after impact began. A follow-up asks why the redis-sidekiq memory alert did not page, and the review says some signals did not reach the right on-call people.

   Remediation

   • Restarted and force-reset the two unresponsive Redis Sidekiq nodes
   • Stopped each Redis Sidekiq VM in turn and resized it to c2-standard-16, doubling memory and CPU
   • Re-enabled AuditEventStreamingWorker instead of deferring it

   Action items

   • Investigate why the redis-sidekiq memory alert did not page
   • Establish the root cause, failure mode and remediation status of the WebHookWorker poison pill
   • Give mitigations that disable safety limits an explicit owner and a deadline for re-evaluating them
   • Stop AuditEvents::AuditEventStreamingWorker from causing recurring incidents
   • Create a runbook for AuditEventStreamingWorker
   • Address PG connection pool saturation

   Lessons

   Broad degradation across web, Git, authentication, CI runners, Sidekiq and the internal API made it hard to settle on one cause and explain it clearly. The customer symptom of pipelines stuck in created was hard to measure in real time, and the dashboards that would have answered it were not obvious to on-call responders. Some alerts did not reach the right on-call people, and hosts that hung on login slowed diagnosis and made changes riskier. Redis-sidekiq under memory saturation forces choices between availability and data loss that are hard to make during an incident and call for guardrails and design changes.

   View case
2. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes

   GitLab2023

   Symptoms

   GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Failover failureHigh error rate

   Trigger

   The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

   Root cause

   Cloud provider outage

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Remediation

   • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary

   Action items

   • Alert when the Sentinel service is down
   • Make merge requests robust to Redis Sidekiq outages

   Lessons

   An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

   View case

By root cause2