Cases2 cases

1 to 2 of 2

1. IL-0290 GitLab.com's Redis Sidekiq cluster ran out of memory and lost quorum, causing 500 errors and a 12 million job backlog

   GitLab2026sev1

   Symptoms

   From 17:12 UTC on 12 May 2026, error rates rose at the same time across GitLab.com's web, Git, API and CI runner services. Users got frequent 500 errors, failed or stuck CI pipelines, authentication failures and unreliable Git access, and a Prometheus Alertmanager alert opened the incident at 17:15.

   High error rateMemory pressureThroughput degradation

   Trigger

   Earlier on 12 May, as a mitigation for a previous incident, INC-10084, a Sidekiq queueing apdex SLO violation on shard catchall-b, jobs of AuditEvents::AuditEventStreamingWorker were deferred. The deferred audit event jobs then piled up in the Redis Sidekiq cluster.

   Root cause

   Capacity exhaustion

   Component

   In-memory cache

   Contributing factor

   Change with a delayed effectPages not deliveredLimited debug access

   Detection

   A Prometheus Alertmanager alert reported the incident at 17:15:10 UTC, three minutes after impact began. A follow-up asks why the redis-sidekiq memory alert did not page, and the review says some signals did not reach the right on-call people.

   Remediation

   • Restarted and force-reset the two unresponsive Redis Sidekiq nodes
   • Stopped each Redis Sidekiq VM in turn and resized it to c2-standard-16, doubling memory and CPU
   • Re-enabled AuditEventStreamingWorker instead of deferring it

   Action items

   • Investigate why the redis-sidekiq memory alert did not page
   • Establish the root cause, failure mode and remediation status of the WebHookWorker poison pill
   • Give mitigations that disable safety limits an explicit owner and a deadline for re-evaluating them
   • Stop AuditEvents::AuditEventStreamingWorker from causing recurring incidents
   • Create a runbook for AuditEventStreamingWorker
   • Address PG connection pool saturation

   Lessons

   Broad degradation across web, Git, authentication, CI runners, Sidekiq and the internal API made it hard to settle on one cause and explain it clearly. The customer symptom of pipelines stuck in created was hard to measure in real time, and the dashboards that would have answered it were not obvious to on-call responders. Some alerts did not reach the right on-call people, and hosts that hung on login slowed diagnosis and made changes riskier. Redis-sidekiq under memory saturation forces choices between availability and data loss that are hard to make during an incident and call for guardrails and design changes.

   View case
2. IL-0261 Discord went fully offline twice as short network losses on Google Cloud sessions nodes wedged Erlang distribution

   Discord2017

   Symptoms

   On 17 and 18 November 2017 nodes of Discord's 'sessions' cluster on Google Cloud Platform lost network connectivity for short periods, 16 times in all. Engineers saw a service impact, a node whose service had died and restarted, and pages for low memory on other sessions nodes; twice the failure cascaded and Discord was completely offline for about 30 minutes.

   Network unreachableMemory pressureService outage

   Trigger

   A software change that Google rolled out to the Google Cloud Platform networking stack on Friday introduced a bug that made VMs in Discord's sessions cluster lose network connectivity for short periods.

   Root cause

   Cloud provider outage

   Component

   Service discovery

   Contributing factor

   Limited debug accessHarmful manual action

   Detection

   At 11:52 PST on 17 November engineers noticed a service impact and escalated it in the incident chat. Later affected nodes showed up as pages to the on-call engineer for low memory on sessions nodes, and on 18 November as a page for a networking incident on sessions-1-11 together with issue reports that the customer experience team escalated from support channels.

   Remediation

   • Forcefully terminated sessions nodes whose Erlang distribution port had failed
   • Escalated the Google support ticket to P1 and notified the Technical Account Manager
   • Restarted several core Discord services on 17 November, making clients reconnect over about 20 minutes
   • Began terminating and rebooting the entire sessions cluster after seeing that rebooted machines did not fail again
   • Disabled message sending globally on 18 November to help the service recover
   • Restarted all members of the sessions cluster
   • Re-enabled message sending after the initial all clear
   • Google rolled back the software change made to its networking stack

   Action items

   • Continue the project to improve how Discord's Erlang clusters handle node failure
   • Expand how upgrades to these services are load tested so the rollout can be completed safely
   • Replace the HTTP library used by Discord's Elixir/Erlang services
   • Fix the instability seen in Erlang's net_kernel component
   • Replace the pieces of the service discovery component that degraded with more performant implementations
   • Work with Google on the issues Discord met while working with Google on this incident
   • Expand the internal procedure for communicating with Google so the progress of P1 issues is tracked

   Lessons

   Discord concluded that performance and stability issues in its own system, in Erlang's net_kernel, the service discovery component and the HTTP library, were what turned short network losses on part of the fleet from isolated issues for some users into two full system outages.

   View case

By root cause2