Cases2 cases

1 to 2 of 2

1. IL-0322 A GCP storage incident degraded the disk under GitLab.com's primary PostgreSQL and caused two bursts of site-wide 503s

   GitLab2026sev1

   Symptoms

   On 2026-06-11 https://gitlab.com returned widespread, intermittent HTTP 503 errors across all major services in two bursts of about 5 to 10 minutes, 06:16 to 06:22 UTC and 07:03 to 07:12 UTC. A Pingdom alert reported the first burst in triage at 06:16:12, about four minutes after impact began at 06:12, and each burst ended on its own.

   Service outageIntermittent timeoutsConnection pool exhaustionLock contention or deadlockReplication lag

   Trigger

   A GCP storage incident degraded the IO of a disk in the `us-east1-b` zone that backed GitLab.com's primary PostgreSQL node.

   Root cause

   Cloud provider outage

   Component

   DatabaseCloud block storage

   Contributing factor

   Single point of failureNo alert on limitNo runbook

   Detection

   A Pingdom alert on the user-facing 503 errors was reported in triage at 06:16:12 UTC. No alert existed for disk IO saturation, pgBouncer pool saturation or LWLock contention.

   Remediation

   • Escalated to Database Operations experts
   • Paused deployments
   • Opened a GCP support case and engaged the GCP TAM, and GCP confirmed the storage incident
   • Prepared a Patroni primary switchover to node 04 in us-east1-d as a contingency, without executing it
   • Held a 3-hour pause-and-review window and resolved the incident after no recurrence

   Action items

   • Add disk IO saturation and queue-depth alerting for primary Patroni database nodes
   • Write a Patroni primary switchover runbook with explicit trigger thresholds
   • Clarify incident state-transition guidance for intermittent, self-resolving issues
   • Decide whether the primary should fail over automatically during disk IO degradation
   • Find a short-term mitigation that limits the blast radius of a single degraded disk until Cells do so
   • Make application database load balancing fail fast or reroute when the primary stops responding
   • Consider alerting on pgBouncer pool saturation and LWLock contention as leading indicators

   Lessons

   A single degraded disk was enough to take down GitLab.com because all primary database traffic goes through one node and requests waited for the Rack timeout instead of failing fast. The leading indicators, disk IO queue depth, pgBouncer pool saturation and LWLock contention, rose before full impact but had no alerts. Without decision criteria for a switchover, responders spent much of an intermittent incident waiting, and it was hard to decide when to move the incident to Fixing or Monitoring.

   View case
2. IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone

   UK Ministry of Justice2022

   Symptoms

   On 11 July 2022 users reported in the #ask-cloud-platform Slack channel that their applications on the Ministry of Justice Cloud Platform were slow some of the time. About 25% of ingress traffic was affected: after an AWS incident the day before, one of the three network interfaces of the 'default' ingress controllers performed slowly, and clients timed out when they tried to connect to one of the Elastic IPs of the Network Load Balancer (NLB) in front of them.

   High latencyIntermittent timeouts

   Trigger

   Some instances of the AWS NLB health checking subsystem were restarted, which set off a latent software race condition in that subsystem.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerIngress controller

   Contributing factor

   –

   Detection

   Users reported slow performance of their applications in the #ask-cloud-platform Slack channel.

   Remediation

   • AWS restarted the NLB health checking subsystem, which refreshed its list of targets and recovered the NLB in the affected Availability Zone

   Action items

   • Raised mitigation tickets after a post-incident review, filed as post-aws-incident issues in the ministryofjustice/cloud-platform repository

   View case

By root cause1