Cases2 cases

1 to 2 of 2

1. IL-0322 A GCP storage incident degraded the disk under GitLab.com's primary PostgreSQL and caused two bursts of site-wide 503s

   GitLab2026sev1

   Symptoms

   On 2026-06-11 https://gitlab.com returned widespread, intermittent HTTP 503 errors across all major services in two bursts of about 5 to 10 minutes, 06:16 to 06:22 UTC and 07:03 to 07:12 UTC. A Pingdom alert reported the first burst in triage at 06:16:12, about four minutes after impact began at 06:12, and each burst ended on its own.

   Service outageIntermittent timeoutsConnection pool exhaustionLock contention or deadlockReplication lag

   Trigger

   A GCP storage incident degraded the IO of a disk in the `us-east1-b` zone that backed GitLab.com's primary PostgreSQL node.

   Root cause

   Cloud provider outage

   Component

   DatabaseCloud block storage

   Contributing factor

   Single point of failureNo alert on limitNo runbook

   Detection

   A Pingdom alert on the user-facing 503 errors was reported in triage at 06:16:12 UTC. No alert existed for disk IO saturation, pgBouncer pool saturation or LWLock contention.

   Remediation

   • Escalated to Database Operations experts
   • Paused deployments
   • Opened a GCP support case and engaged the GCP TAM, and GCP confirmed the storage incident
   • Prepared a Patroni primary switchover to node 04 in us-east1-d as a contingency, without executing it
   • Held a 3-hour pause-and-review window and resolved the incident after no recurrence

   Action items

   • Add disk IO saturation and queue-depth alerting for primary Patroni database nodes
   • Write a Patroni primary switchover runbook with explicit trigger thresholds
   • Clarify incident state-transition guidance for intermittent, self-resolving issues
   • Decide whether the primary should fail over automatically during disk IO degradation
   • Find a short-term mitigation that limits the blast radius of a single degraded disk until Cells do so
   • Make application database load balancing fail fast or reroute when the primary stops responding
   • Consider alerting on pgBouncer pool saturation and LWLock contention as leading indicators

   Lessons

   A single degraded disk was enough to take down GitLab.com because all primary database traffic goes through one node and requests waited for the Rack timeout instead of failing fast. The leading indicators, disk IO queue depth, pgBouncer pool saturation and LWLock contention, rose before full impact but had no alerts. Without decision criteria for a switchover, responders spent much of an intermittent incident waiting, and it was hard to decide when to move the incident to Fixing or Monitoring.

   View case
2. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage

   PlanetScale2025

   Symptoms

   At 7:13 UTC on 20 October 2025, PlanetScale's continuous tests in production started failing because they could not create new database branches. The service that creates, resizes and configures database branches was unavailable, a near-total control plane outage, while existing database branches kept their capacity and connectivity.

   Service outageScale-up failureNetwork unreachableHigh error rateReplication lagDNS resolution failures

   Trigger

   The incident started with a DNS misconfiguration at one of PlanetScale's service providers. An Amazon DynamoDB outage impacted AWS STS, and several hours of capacity constraints and network instability in AWS us-east-1 followed.

   Root cause

   Cloud provider outage

   Component

   DatabaseDatabase replicationDatabase proxy and connection poolerObject storage

   Contributing factor

   –

   Detection

   PlanetScale's continuous testing in production started failing and alerted engineers at 7:13 UTC. At 10:05 UTC an alert fired for a Kubernetes operator in a large single-tenant customer installation exhausting its resources.

   Remediation

   • Disallowed new databases in us-east-1 and made us-east-2 the default region for new databases
   • Delayed new backups and cancelled pending backups that were waiting for an EC2 instance
   • Advised PlanetScale Managed customers with vtgate autoscaling to shed load
   • Paused draining and terminating EC2 instances older than 30 days
   • Kept vacant EC2 instances for reuse instead of terminating them
   • Bin-packed vtgate processes more tightly for customers with autoscaling
   • Sent manual reparent requests to move primaries to healthier availability zones
   • Restarted the edge load balancer and vtgate processes that did not recover after the partitions healed

   Action items

   • Understand and become resilient to the failure modes of SaaS dependencies for CI/CD, SSO, web application hosting and incident communication
   • Investigate more ambitious ways to reduce runtime dependence on internal and AWS services
   • Look into using all six us-east-1 availability zones to tolerate zonal outages and partitions

   Lessons

   Keeping the control plane separate from the data plane meant the control plane outage did not affect customer databases, and redundancy, automated failover and zonal traffic routing limited the effect of the network partitions. Network partitions are hard to reason about, test and tolerate: with three availability zones a system survives losing one zone only if the other two can still reach each other reliably.

   View case

By root cause1