Incident Library

IL-0218 GitLab · 2017-01-31databaseazure

GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hours

GitLab.com ran one PostgreSQL primary, db1.cluster.gitlab.com, and one hot-standby secondary, db2.cluster.gitlab.com, used only for failover, so a single database carried all the load; GitLab had already had several other problems from this single point of failure. On 31 January an engineer was setting up PostgreSQL servers in staging to try pgpool-II, and around 17:20 UTC took an LVM snapshot of the production database for staging. From about 19:00 UTC database load rose from what was suspected to be spam, together with a background job removing a GitLab employee whose account a troll had reported for abuse and which had been scheduled for removal by mistake. Around 23:00 UTC the secondary's replication failed because GitLab.com did not use WAL archiving and the primary had removed the WAL segments the secondary needed. Re-synchronising it by hand, which was neither automated nor properly documented, led to the hung pg_basebackup runs and the max_wal_senders and max_connections changes, and around 23:30 UTC to the accidental wipe of about 300 GB on the primary.

The recovery procedures then failed one after another. The pg_dump backups uploaded to S3 every 24 hours had been failing silently: the job ran on an application server without a PostgreSQL data directory, so the Omnibus package defaulted to pg_dump 9.2, which errors against a 9.6 database, and the failure emails were rejected for lack of DMARC. Azure disk snapshots, which can take days to restore across storage accounts, had not been enabled for the database servers on the assumption that the other backups were enough. Replication was meant for failover, not disaster recovery, and the secondary's data had already been wiped. Nobody owned the backup procedure, so nobody tested it.

GitLab restored from the LVM snapshot taken about 6 hours before the outage, losing less than the 24-hour snapshot would have. Copying the PostgreSQL data directory from the staging host, on Azure classic storage without Premium Storage throttled to around 60Mbps, took around 18 hours. On 1 February at 17:00 UTC the database was restored without webhooks, which the staging copy procedure removes; webhooks were restored from a separate database built from the snapshot, and the final steps were finished around 18:00 UTC. Database sequences were incremented by 100,000 so earlier IDs could not be reused, GitLab.com was re-enabled gradually, and projects created after 17:20 UTC were rebuilt in basic form from their Git repositories on the NFS cluster. GitLab kept public notes in a Google document and streamed the recovery on YouTube.

Database changes such as projects, comments, user accounts, issues and snippets made after 17:20 UTC on 31 January were lost; GitLab gives the end of that window as 00:00 UTC at the top of the post and as 23:30 UTC in its data loss section, and estimated roughly 5,000 projects, 5,000 comments and 700 new user accounts were affected. Git repositories and wikis were stored separately and not lost.

Severity
–
Scope
GitLab.com unavailable for about 18 hours; database changes made after 17:20 UTC on 31 January, such as projects, comments, user accounts, issues and snippets, were lost (GitLab gives the end of the window as 00:00 UTC in its summary and 23:30 UTC in its data loss section); Git repositories and wikis were unavailable but not lost
Users affected
GitLab.com users; roughly 5,000 projects, 5,000 comments and 700 new user accounts lost; GitLab Enterprise, GitHost and self-managed GitLab CE users not affected
Region
–
Time to detect
–
Time to resolve
18h

Symptoms

From around 19:00 UTC on 31 January 2017, load on GitLab.com's PostgreSQL database rose, which GitLab suspected was spam, and many users could not post comments on issues and merge requests. Around 23:00 UTC the replication of the PostgreSQL secondary fell behind and then failed, because WAL segments it still needed had already been removed from the primary.

Rebuilding the secondary meant emptying its data directory and copying the database over with pg_basebackup. pg_basebackup hung without meaningful output even with --verbose, and after several tries reported that it could not connect because the master did not have enough replication connections, a limit set by max_wal_senders. When max_wal_senders was raised from its default of 3 to 32, PostgreSQL refused to restart, complaining that too many semaphores were being created, because max_connections had been set to 8000 almost a year earlier. With max_connections at 2000 PostgreSQL restarted, but pg_basebackup still did not start replicating, and strace only showed it hanging in a poll call. pg_basebackup normally waits silently until the primary starts sending data, but neither GitLab's runbooks nor the pg_basebackup documentation said so clearly.

Around 23:30 UTC an engineer, thinking that earlier attempts had left files behind, wiped the PostgreSQL data directory again, but on the primary instead of the secondary. The engineer stopped the command a second or two later, after about 300 GB had already been removed, and GitLab.com went down with data now wiped from both hosts.

None of the backups could be used quickly. The S3 bucket for the daily pg_dump backups was empty, because the backup job ran pg_dump 9.2 against PostgreSQL 9.6 and failed, and the cron failure emails were rejected by the receiving mail server because they were not signed with DMARC. Azure disk snapshots were not enabled for the database servers. The only usable copies were LVM snapshots made for the staging environment, one almost 24 hours old and one taken by hand about 6 hours before the outage, and copying it from the staging disks, throttled to around 60Mbps, took about 18 hours.

Database changes such as projects, comments, user accounts, issues and snippets made after 17:20 UTC on 31 January were lost. GitLab gives the end of that window as 00:00 UTC at the top of the post and as 23:30 UTC in its data loss section. It estimated roughly 5,000 projects, 5,000 comments and 700 new user accounts were affected. Git repositories and wikis were unavailable during the outage but not lost.

The public monitoring website dashboards.gitlab.com could not handle the load from users during the outage.

Symptom Database overloadReplication lagData lossService outage

Root cause

Root cause Operator error

Component DatabaseDatabase replicationDatabase backups and snapshotsCloud block storage
Contributing factor Untested backupSingle point of failureNo runbookPages not deliveredUnverified assumptionUnreviewed defaults
Trigger
While trying to restore replication to the secondary, an engineer ran the removal of the PostgreSQL data directory on the primary, db1.cluster.gitlab.com, believing it was the secondary.

Detection

The engineer who ran the deletion on the primary noticed the mistake and stopped it a second or two later.

Remediation

Action items

Lessons

GitLab split the incident into two problems: GitLab.com went down because the primary's data directory was removed by mistake during a manual, undocumented replication rebuild, and recovery took over 18 hours because the backups had failed silently, disk snapshots were not enabled and the only copy sat on slow staging disks. The backup procedure was never tested because no one owned it. GitLab chose to focus on fast, regularly practised recovery and on making it obvious which host an engineer is on, rather than on blocking commands, since a safer rm only protects against one of many ways to lose data.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database replication, Database backups and snapshots among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.