Cases2 cases

1 to 2 of 2

1. IL-0322 A GCP storage incident degraded the disk under GitLab.com's primary PostgreSQL and caused two bursts of site-wide 503s

   GitLab2026sev1

   Symptoms

   On 2026-06-11 https://gitlab.com returned widespread, intermittent HTTP 503 errors across all major services in two bursts of about 5 to 10 minutes, 06:16 to 06:22 UTC and 07:03 to 07:12 UTC. A Pingdom alert reported the first burst in triage at 06:16:12, about four minutes after impact began at 06:12, and each burst ended on its own.

   Service outageIntermittent timeoutsConnection pool exhaustionLock contention or deadlockReplication lag

   Trigger

   A GCP storage incident degraded the IO of a disk in the `us-east1-b` zone that backed GitLab.com's primary PostgreSQL node.

   Root cause

   Cloud provider outage

   Component

   DatabaseCloud block storage

   Contributing factor

   Single point of failureNo alert on limitNo runbook

   Detection

   A Pingdom alert on the user-facing 503 errors was reported in triage at 06:16:12 UTC. No alert existed for disk IO saturation, pgBouncer pool saturation or LWLock contention.

   Remediation

   • Escalated to Database Operations experts
   • Paused deployments
   • Opened a GCP support case and engaged the GCP TAM, and GCP confirmed the storage incident
   • Prepared a Patroni primary switchover to node 04 in us-east1-d as a contingency, without executing it
   • Held a 3-hour pause-and-review window and resolved the incident after no recurrence

   Action items

   • Add disk IO saturation and queue-depth alerting for primary Patroni database nodes
   • Write a Patroni primary switchover runbook with explicit trigger thresholds
   • Clarify incident state-transition guidance for intermittent, self-resolving issues
   • Decide whether the primary should fail over automatically during disk IO degradation
   • Find a short-term mitigation that limits the blast radius of a single degraded disk until Cells do so
   • Make application database load balancing fail fast or reroute when the primary stops responding
   • Consider alerting on pgBouncer pool saturation and LWLock contention as leading indicators

   Lessons

   A single degraded disk was enough to take down GitLab.com because all primary database traffic goes through one node and requests waited for the Rack timeout instead of failing fast. The leading indicators, disk IO queue depth, pgBouncer pool saturation and LWLock contention, rose before full impact but had no alerts. Without decision criteria for a switchover, responders spent much of an intermittent incident waiting, and it was hard to decide when to move the incident to Fixing or Monitoring.

   View case
2. IL-0488 Kit's RDS MySQL write latency rose 100x and running threads jumped from 9 to 900, backing up Sidekiq job queues

   Kit2022

   Symptoms

   At 5:20 am EST on December 24, 2022, Kit's infrastructure team got its first page because the number of MySQL threads running on its RDS MySQL primary in us-east-2 had jumped 100x, from 9 to 900, at a time of day when Kit's traffic is very low. Sidekiq queues were backing up.

   Database overloadThroughput degradation

   Trigger

   An EBS interruption on the standby of the Multi-AZ RDS instance, reported by AWS as an LSE at 02:12:47 UTC on 2022-12-24, started a replaceSecondaryHostAndVolumesTask that replaced the secondary host and volumes.

   Root cause

   Cloud provider outage

   Component

   DatabaseCloud block storageDatabase replication

   Contributing factor

   Ignored early signal

   Detection

   A page fired at 5:20 am EST because MySQL threads running jumped from 9 to 900. An AWS RSS feed alert in Slack about degraded EC2 and EBS in use2-az1 had arrived the evening before and was not acted on.

   Remediation

   • Scaled the RDS storage from 12.5 TB to 13.75 TB, which put the instance through "optimizing storage" and brought write latency back to 1-2 ms

   Lessons

   Multi-AZ RDS keeps data consistent and durable, but synchronous replication means each write also waits for the standby, so a degraded EBS volume on the standby while it is being replaced slows writes on the primary. Large databases recover more slowly from such events than small ones. A single-AZ instance would not have been hit by a standby problem, but would have suffered hours of downtime if its own zone failed, and that trade-off has to be weighed when deploying production RDS instances.

   View case

By root cause1