Cases2 cases

1 to 2 of 2

1. IL-0322 A GCP storage incident degraded the disk under GitLab.com's primary PostgreSQL and caused two bursts of site-wide 503s

   GitLab2026sev1

   Symptoms

   On 2026-06-11 https://gitlab.com returned widespread, intermittent HTTP 503 errors across all major services in two bursts of about 5 to 10 minutes, 06:16 to 06:22 UTC and 07:03 to 07:12 UTC. A Pingdom alert reported the first burst in triage at 06:16:12, about four minutes after impact began at 06:12, and each burst ended on its own.

   Service outageIntermittent timeoutsConnection pool exhaustionLock contention or deadlockReplication lag

   Trigger

   A GCP storage incident degraded the IO of a disk in the `us-east1-b` zone that backed GitLab.com's primary PostgreSQL node.

   Root cause

   Cloud provider outage

   Component

   DatabaseCloud block storage

   Contributing factor

   Single point of failureNo alert on limitNo runbook

   Detection

   A Pingdom alert on the user-facing 503 errors was reported in triage at 06:16:12 UTC. No alert existed for disk IO saturation, pgBouncer pool saturation or LWLock contention.

   Remediation

   • Escalated to Database Operations experts
   • Paused deployments
   • Opened a GCP support case and engaged the GCP TAM, and GCP confirmed the storage incident
   • Prepared a Patroni primary switchover to node 04 in us-east1-d as a contingency, without executing it
   • Held a 3-hour pause-and-review window and resolved the incident after no recurrence

   Action items

   • Add disk IO saturation and queue-depth alerting for primary Patroni database nodes
   • Write a Patroni primary switchover runbook with explicit trigger thresholds
   • Clarify incident state-transition guidance for intermittent, self-resolving issues
   • Decide whether the primary should fail over automatically during disk IO degradation
   • Find a short-term mitigation that limits the blast radius of a single degraded disk until Cells do so
   • Make application database load balancing fail fast or reroute when the primary stops responding
   • Consider alerting on pgBouncer pool saturation and LWLock contention as leading indicators

   Lessons

   A single degraded disk was enough to take down GitLab.com because all primary database traffic goes through one node and requests waited for the Rack timeout instead of failing fast. The leading indicators, disk IO queue depth, pgBouncer pool saturation and LWLock contention, rose before full impact but had no alerts. Without decision criteria for a switchover, responders spent much of an intermittent incident waiting, and it was hard to decide when to move the incident to Fixing or Monitoring.

   View case
2. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes

   GitLab2023

   Symptoms

   GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Failover failureHigh error rate

   Trigger

   The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

   Root cause

   Cloud provider outage

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Remediation

   • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary

   Action items

   • Alert when the Sentinel service is down
   • Make merge requests robust to Redis Sidekiq outages

   Lessons

   An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

   View case

By root cause1