Cases1 case
1 to 1 of 1
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab2021- Symptoms
At 23:38 UTC on 2021-01-16 GitLab.com error rates spiked, and a minute later multiple alerts for service disruption arrived, two minutes after GCP disk snapshots had been started on all Patroni database nodes. Some requests that needed database writes returned 5xx errors for about four to five minutes, while page refreshes checked during the incident responded normally.
- Trigger
During the change production#2648, at 23:36 UTC an engineer started GCP disk snapshots on all Patroni nodes at the same time, as a safety step before a change to the filesystem, believing the snapshot was a transparent background operation through the GCP API.
- Root cause
- Component
- Contributing factor
- Detection
A flood of alerts about downstream impact; multiple service disruption alerts arrived at 23:39 UTC, a minute after the error spikes began. The incident was declared in Slack at 23:56.
- Remediation
- Patroni invalidated patroni-06's lock and promoted patroni-03 as the new primary
- Cancelled the snapshot requests in two of the three zones
- Action items
- Update the Patroni and PostgreSQL runbooks to clarify GCP snapshot best practices
- Add a checklist for change management reviewers and approvers to the runbooks, referenced in the handbook
- Schedule a call with Google about IO sensitivities
- Lessons
Do not take a GCP snapshot of a running PostgreSQL leader. Technical reviewers have to review every change or addition to a change request's description before a manager or director approves it. The Datastores team should explain in detail how backups and snapshots work on the production database. No action belonging to a change should start before the change's scheduled start.
This case involves Cloud block storage and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.