IL-0290 GitLab · 2026-05-12databasegcp
GitLab.com's Redis Sidekiq cluster ran out of memory and lost quorum, causing 500 errors and a 12 million job backlog
GitLab.com runs its Sidekiq background jobs on Redis Sidekiq clusters; the catchall-b shard ran on three VMs, redis-sidekiq-catchall-b-01-db-gprd, -02 and -03. Earlier on 12 May, responders to INC-10084, a Sidekiq queueing apdex SLO violation on shard catchall-b, deferred the jobs of AuditEvents::AuditEventStreamingWorker. The deferred audit event jobs accumulated in Redis until they had taken all available memory on the cluster. The memory exhaustion led to kernel errors and made several Redis nodes fail, leaving one healthy node that could not keep cluster quorum. From 17:12 UTC every major GitLab.com service had raised error rates, authentication failures and delayed jobs, and the incident was raised to Severity 1 at 17:32.
Symptoms across many services and hosts that hung on SSH login slowed the diagnosis. Once the catchall-b nodes were confirmed unhealthy, engineers restarted and force-reset the two unresponsive nodes; all three were up by 18:00, but errors came back. Engineers then stopped each Redis VM in turn and resized it to the c2-standard-16 machine type, doubling its memory and CPU, starting with catchall-b-01 at about 18:10 and stopping catchall-b-03 at 18:25. By 18:40 all three nodes were upgraded and error rates fell, and impact ended at 18:43:44 UTC, 1 hour 31 minutes after it began. AuditEventStreamingWorker was re-enabled instead of being deferred, the backlog dropped from its peak of 12.2 million jobs to under 7 million by 19:10, and the incident was resolved at 21:30. Responders held back from irreversible options such as flushing queues, and the review notes that Redis-sidekiq under memory saturation forced hard choices between availability and data loss.
The review names AuditEvents::AuditEventStreamingWorker as a recurring contributing factor, counting at least 4 prior incidents since March 2025 that grew in severity each time. It lists INC-310 (S2, 24 March 2025, PGBouncer saturation), INC-311 (S2, a repeat the next day), INC-489 (S2, 15 April 2025, a bad streaming destination) and then this S1 incident. During the incident a responder also noted that this was the second S2 involving the worker that month, after INC-8169, and that there had been an S3 in December.
- Severity
- sev1
- Scope
- All major GitLab.com services, including web, Git, API, CI runners and Sidekiq: more than 1 million requests affected, a Sidekiq backlog of over 12 million jobs and an estimated more than 10,000 pipelines affected
- Users affected
- All GitLab.com customers active during the impact period
- Region
- –
- Time to detect
- 3m
- Time to resolve
- 1h 31m
Symptoms
From 17:12 UTC on 12 May 2026, error rates rose at the same time across GitLab.com's web, Git, API and CI runner services. Users got frequent 500 errors, failed or stuck CI pipelines, authentication failures and unreliable Git access, and a Prometheus Alertmanager alert opened the incident at 17:15.
The platform triage dashboard showed many services erroring at once, with CI runners at up to 5% errors and the git workhorse API and the web service also affected. On the Redis Sidekiq hosts of the catchall-b shard, the kernel logged "INFO: task kcompactd0:67 blocked for more than 120 seconds." and the redis-rdb-backup.service unit failed on redis-sidekiq-catchall-b-01-db-gprd. On redis-sidekiq-catchall-b-02-db-gprd, systemd reported "Found left-over process 2516163 (redis-server) in control group while starting unit", redis_exporter logged "Couldn't connect to redis instance (redis://localhost:6379)", and Teleport logged "SLOW TRANSACTION" warnings and name lookups ending in "i/o timeout". SSH logins to the hosts hung.
Responders first looked at Redis connectivity, then at evidence that seemed to point to DNS service discovery or a wider disruption in the US East region, and asked for outside DNS expertise. No rollout was running in production; the only recent change was a registry rollout in staging. Request rates on the dashboards were flat or falling, which ruled out a traffic surge.
By 18:00 the two unresponsive Redis nodes had been restarted and reset, and all three nodes, O1, O2 and O3, were up with O2 as primary, but the nodes started throwing errors again right away. The Sidekiq queue backlog peaked at 12.2 million jobs, pipeline creation stayed near zero for long stretches and pipelines sat stuck in created, with more than 10,000 pipelines estimated to be affected. Responders found the "Pipelines stuck in created" symptom hard to measure and confirm while it was happening.
Symptom High error rateMemory pressureThroughput degradation
Root cause
Root cause Capacity exhaustion
- Trigger
- Earlier on 12 May, as a mitigation for a previous incident, INC-10084, a Sidekiq queueing apdex SLO violation on shard catchall-b, jobs of AuditEvents::AuditEventStreamingWorker were deferred. The deferred audit event jobs then piled up in the Redis Sidekiq cluster.
Detection
A Prometheus Alertmanager alert reported the incident at 17:15:10 UTC, three minutes after impact began. A follow-up asks why the redis-sidekiq memory alert did not page, and the review says some signals did not reach the right on-call people.
Remediation
- Restarted and force-reset the two unresponsive Redis Sidekiq nodes
- Stopped each Redis Sidekiq VM in turn and resized it to c2-standard-16, doubling memory and CPU
- Re-enabled AuditEventStreamingWorker instead of deferring it
Action items
- Investigate why the redis-sidekiq memory alert did not page
- Establish the root cause, failure mode and remediation status of the WebHookWorker poison pill
- Give mitigations that disable safety limits an explicit owner and a deadline for re-evaluating them
- Stop AuditEvents::AuditEventStreamingWorker from causing recurring incidents
- Create a runbook for AuditEventStreamingWorker
- Address PG connection pool saturation
Lessons
Broad degradation across web, Git, authentication, CI runners, Sidekiq and the internal API made it hard to settle on one cause and explain it clearly. The customer symptom of pipelines stuck in created was hard to measure in real time, and the dashboards that would have answered it were not obvious to on-call responders. Some alerts did not reach the right on-call people, and hosts that hung on login slowed diagnosis and made changes riskier. Redis-sidekiq under memory saturation forces choices between availability and data loss that are hard to make during an incident and call for guardrails and design changes.
Sources
References
Related cases
IL-0432AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limitHello Inc. · 2026same root cause · shared In-memory cache
IL-058129CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week saleMUSINSA · 2025same root cause · shared In-memory cache
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared In-memory cache
IL-0067Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restartBC Government · 2025 · sev1same root cause · shared In-memory cache
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared In-memory cache
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared In-memory cache
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared In-memory cache
IL-0453Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forklessHyperconnect · 2022same root cause · shared In-memory cache
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared In-memory cache
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared In-memory cache
IL-0520logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessionscloud.gov · 2016same root cause · shared In-memory cache
IL-0296Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customersTransloadit · 2016same root cause · shared In-memory cache
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0290
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0290/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.