Incident Library

IL-0322 GitLab · 2026-06-11databasegcp

A GCP storage incident degraded the disk under GitLab.com's primary PostgreSQL and caused two bursts of site-wide 503s

All primary database traffic for GitLab.com goes through a single primary node, patroni-main-v17-01-db-gprd in project gitlab-production. A confirmed GCP storage incident affected a disk backing that node in `us-east1-b`, and the node suffered severe IO contention, seen as disk IO queue depth spikes, LWLock contention, replication lag and pgBouncer connection pool saturation. Calls to the primary were blocked, and because the application's database load balancing did not shed or reroute requests when the primary stopped responding, requests piled up until the 60-second Rack timeout instead of failing fast. That turned one degraded disk into two short site-wide outages; both ended without manual remediation.

The incident was raised to Severity 1 at 06:17:42. Responders paused deployments, engaged the GCP TAM and opened a support case, and GCP confirmed the storage incident; the root cause was confirmed at 12:15:57. Failover depended on a human decision with no agreed threshold, so the team prepared a Patroni switchover to another zone as a contingency, weighing its effect on write throughput, and paused the incident for a 3-hour review at 15:19:42; no recurrence was seen and the platform has been fully operational since 07:25 UTC. There was no direct alerting on disk IO saturation, pgBouncer pool saturation or LWLock contention, which were the leading indicators. The review ties the missing fail-fast behaviour to learnings from the earlier incident #18269.

Severity
sev1
Scope
Web, frontend, API, git, websockets, web-pages and CI runners on https://gitlab.com returned HTTP 503 errors during 06:16-06:22 and 07:03-07:12 UTC
Users affected
All external customers and users of https://gitlab.com
Region
GCP us-east1-b (zone of the degraded disk)
Time to detect
4m
Time to resolve
1h

Symptoms

On 2026-06-11 https://gitlab.com returned widespread, intermittent HTTP 503 errors across all major services in two bursts of about 5 to 10 minutes, 06:16 to 06:22 UTC and 07:03 to 07:12 UTC. A Pingdom alert reported the first burst in triage at 06:16:12, about four minutes after impact began at 06:12, and each burst ended on its own.

Web, frontend, API, git, websockets, web-pages and CI runners were all affected. Requests timed out at the 60-second Rack limit because network calls to Postgres and Redis were blocked. HAProxy 5xx errors spiked in both windows, peaking at 06:18 to 06:20 and 07:06 to 07:08. On the primary database instance patroni-main-v17-01-db-gprd, disk IO queue depth jumped to about 219 at 06:16 and to about 353 to 405 at 07:00 to 07:02, and LWLock contention, `pg_wait_sampling_buffer_mapping`, replication lag and pgBouncer pool saturation in both the sync and async pools spiked during the windows.

After the first burst the initial assessment pointed to a temporary external networking issue, and the incident went to Monitoring at 06:30:06. Cloudflare data showed no DDoS activity and no unusual request volume, and slow queries were ruled out early. When the second burst began at 07:03, pgBouncer pool saturation across the sync and async pools pointed to the database layer, and the incident was escalated to Database Operations at 07:11:35. Responders found the `pg_wait_sampling_buffer_mapping` spikes at 07:36:45, the LWLock contention at 08:00:22 and the two disk IO queue depth spikes on the primary at 08:13:27, then contacted the GCP TAM and opened a GCP support case at 08:56:29.

Because the errors came and went, much of the response was spent watching and waiting to see whether they would return. There was no agreed threshold for when to fail over, and with a widespread cloud provider problem the team could not confirm in advance which zone was healthy enough to switch to. A Patroni primary switchover to node `04` in `us-east1-d` was prepared at 12:29:08 but not executed.

Symptom Service outageIntermittent timeoutsConnection pool exhaustionLock contention or deadlockReplication lag

Root cause

Root cause Cloud provider outage

Component DatabaseCloud block storage
Contributing factor Single point of failureNo alert on limitNo runbook
Trigger
A GCP storage incident degraded the IO of a disk in the `us-east1-b` zone that backed GitLab.com's primary PostgreSQL node.

Detection

A Pingdom alert on the user-facing 503 errors was reported in triage at 06:16:12 UTC. No alert existed for disk IO saturation, pgBouncer pool saturation or LWLock contention.

Remediation

Action items

Lessons

A single degraded disk was enough to take down GitLab.com because all primary database traffic goes through one node and requests waited for the Rack timeout instead of failing fast. The leading indicators, disk IO queue depth, pgBouncer pool saturation and LWLock contention, rose before full impact but had no alerts. Without decision criteria for a switchover, responders spent much of an intermittent incident waiting, and it was hard to decide when to move the incident to Fixing or Monitoring.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Cloud block storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.