Cases4 cases

1 to 4 of 4

1. IL-0527 OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc error

   OpenStreetMap Foundation2025

   Symptoms

   At 21:40 UTC on 25 June 2025, Alertmanager alerted the OpenStreetMap operations team: minutely replication feed delayed - 6m 2s. Since 21:33 UTC every run of the osmdbt timer that produces the minutely replication diffs had failed with ERROR: invalid memory alloc request size 1243650064.

   Replication lag

   Root cause

   Software defect

   Component

   Database replicationDatabase

   Contributing factor

   Pages not delivered

   Detection

   Alertmanager sent the first alert at 21:40 UTC, minutely replication feed delayed - 6m 2s, and several more followed, but none was a pager or SMS alert. A community report followed at 00:37 UTC and a social media report at 01:13 UTC.

   Remediation

   • Stopped the osmdbt timer
   • Put OpenStreetMap.org into a read-only maintenance window to suppress edits
   • Ran osmdbt by hand with fakelog to skip ahead of the PostgreSQL replication condition
   • Restarted the osmdbt timer for minutely diffs
   • Ended the read-only maintenance window

   Action items

   • Upgrade PostgreSQL to 17.x
   • Raise an issue with the PostgreSQL developers

   Lessons

   Communication with the osmdbt developer was crucial for working out the workaround, and the operations team's understanding of osmdbt was limited.

   View case
2. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-only

   Wikimedia Foundation2019

   Symptoms

   At 18:40 UTC db1075, the primary database of section s3, stopped answering, and all s3 wikis went read-only because no writes could reach it. Reads kept working from the replicas. At 18:42:45 Icinga reported "PROBLEM - Host db1075 is DOWN: PING CRITICAL - Packet loss = 100%".

   Writes rejectedReplication lagHigh latency

   Trigger

   The RAID battery backup unit (BBU) of db1075, the s3 primary database host, failed and the whole host crashed.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase replication

   Contributing factor

   Pages not deliveredHarmful manual action

   Detection

   Icinga reported db1075 HOST DOWN on IRC at 18:42:45, but that alert does not page; replica Slave IO alerts followed, users reported problems on #wikimedia-operations, and the first SMS page went out at 18:56. Volunteers and staff noticed the failure before the alerts did.

   Remediation

   • Power-cycled db1075 from the iDRAC after finding the BBU problem
   • Let MySQL finish InnoDB recovery on db1075
   • Removed read_only=ON from db1075 by hand
   • Ran puppet by hand a second time so pt-heartbeat started

   Action items

   • Implement or refactor a script that moves replicas when the primary is unavailable
   • Fix the MediaWiki heartbeat model so pt-heartbeat does not use a super-user, avoids a single point of failure and follows the real primary without depending on puppet
   • Decide what to do with the batch of hosts that already had BBU issues
   • Buy a new BBU for db1075
   • Remove db1075 from the primary role
   • Address MediaWiki spam while the database is read-only or the primary is unavailable
   • Make primary database hosts page on HOST DOWN
   • Track hardware errors better in Netbox

   Lessons

   A BBU failure should not crash a whole host, yet it had happened before on HP hosts. HOST DOWN on a primary database should page instead of only alerting on IRC. Starting pt-heartbeat through puppet races with MySQL startup, and the unnoticed failure kept lag reported and made the outage 8 minutes longer.

   View case
3. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hours

   GitLab2017

   Symptoms

   From around 19:00 UTC on 31 January 2017, load on GitLab.com's PostgreSQL database rose, which GitLab suspected was spam, and many users could not post comments on issues and merge requests. Around 23:00 UTC the replication of the PostgreSQL secondary fell behind and then failed, because WAL segments it still needed had already been removed from the primary.

   Database overloadReplication lagData lossService outage

   Trigger

   While trying to restore replication to the secondary, an engineer ran the removal of the PostgreSQL data directory on the primary, db1.cluster.gitlab.com, believing it was the secondary.

   Root cause

   Operator error

   Component

   DatabaseDatabase replicationDatabase backups and snapshotsCloud block storage

   Contributing factor

   Untested backupSingle point of failureNo runbookPages not deliveredUnverified assumptionUnreviewed defaults

   Detection

   The engineer who ran the deletion on the primary noticed the mistake and stopped it a second or two later.

   Remediation

   • Raised max_wal_senders from 3 to 32 and lowered max_connections from 8000 to 2000 so PostgreSQL would restart
   • Stopped the deletion on the primary a second or two after noticing it
   • Restored the database from the LVM snapshot taken about 6 hours before the outage by copying the PostgreSQL data directory from staging
   • Restored the webhooks from a separate database built from the snapshot and imported them as a SQL dump
   • Incremented all database sequences by 100,000 so IDs used before the outage could not be reused
   • Re-enabled GitLab.com gradually
   • Rebuilt projects created after 17:20 UTC in basic form from their Git repositories on the NFS cluster

   Action items

   • Change the shell prompt (PS1) on all hosts to show more clearly which host and environment is in use
   • Monitor backups with Prometheus
   • Set PostgreSQL's max_connections to a sane value
   • Investigate point-in-time recovery and continuous archiving for PostgreSQL
   • Take LVM snapshots of the production databases every hour
   • Take Azure disk snapshots of the production databases
   • Move staging to the ARM environment
   • Recover the production replicas
   • Test recovery of PostgreSQL database backups automatically
   • Improve the PostgreSQL replication documentation and runbooks
   • Investigate pgbarman for PostgreSQL backups
   • Investigate WAL-E for database backup and realtime replication
   • Build a streaming database restore
   • Assign an owner for data durability
   • Set up multiple secondaries and balance load among them with pgpool-II
   • Provide a tool for developers to run and revert Rails migrations on staging
   • Build owned recovery procedures for the rest of the GitLab.com infrastructure, not only the database
   • Publish a backup monitoring dashboard and add more backup data to it
   • Improve the abuse reporting and response system so removal for spam does not hard delete
   • Redact engineers' names in future public incident documents

   Lessons

   GitLab split the incident into two problems: GitLab.com went down because the primary's data directory was removed by mistake during a manual, undocumented replication rebuild, and recovery took over 18 hours because the backups had failed silently, disk snapshots were not enabled and the only copy sat on slow staging disks. The backup procedure was never tested because no one owned it. GitLab chose to focus on fast, regularly practised recovery and on making it obvious which host an engineer is on, rather than on blocking commands, since a safer rm only protects against one of many ways to lose data.

   View case
4. IL-0333 A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com down

   Buildkite2016

   Symptoms

   On 22 August 2016 at 17:21 UTC buildkite.com went offline: nobody could log in, view build logs or read the documentation, although builds kept running and GitHub and Bitbucket pull request statuses kept updating. The Buildkite team in Melbourne was asleep and woke at 21:00 UTC, almost 4 hours later, to emails, tweets and Slack messages saying Buildkite was down.

   Service outageDatabase overloadProbe failuresCloud API rate limit exhausted

   Trigger

   Over a weekend maintenance window, Buildkite downgraded its m4.10xlarge Multi-AZ RDS PostgreSQL database to an r3.2xlarge Multi-AZ instance to cut its AWS bill, and moved to a centralised pgbouncer to pool connections. The first US weekday peak after the change pushed the new instance's CPU to its maximum at about 16:00 UTC.

   Root cause

   Capacity exhaustion

   Component

   DatabaseCloud load balancerLiveness and readiness probesNode launch template

   Contributing factor

   Unverified assumptionHealth check tied to a dependencyAutomation amplificationCircular dependencyPages not deliveredOverlapping failures

   Detection

   New Relic, Pingdom, Datadog, Calibre and CloudWatch raised PagerDuty alerts, but the on-call phone call never came and only silent push notifications arrived. The team noticed at 21:00 UTC when they woke to emails, tweets and Slack messages from users.

   Remediation

   • Moved servers from the Agent API to the dashboard ELB to bring buildkite.com back online
   • Switched the Auto Scaling Groups to EC2 health checks and locked in desired counts
   • Upgraded the database to an instance type that could handle the load, using the aws-cli
   • Built a new AMI that did not depend on buildkite.com/_secret/version and rolled out servers that could self-deploy
   • Removed the centralised pgbouncer and went back to pgbouncer on each host

   Action items

   • Plan load testing after significant infrastructure changes without affecting other users of the system
   • Make the ELB health check return OK when the app answers, without checking database connectivity
   • Consider a separate health check endpoint for Pingdom that checks the database and other dependencies
   • Store the latest deployed revision as a file on S3 for newly launched servers to self-deploy from
   • Test by hand that PagerDuty calls reach every on-call team member
   • Investigate regular company-wide tests of everyone's paging setup
   • Ask PagerDuty about calling members when they go on call or requiring an immediate phone rule

   Lessons

   Buildkite listed six lessons. Watch AWS credits, because billing reports show $0 while credits are spent, so the real cost is hidden and big changes end up rushed. Load test after significant infrastructure changes. Think about what a health check should do and what happens when every server behind an ELB fails it. Do not make new servers look up the deployed revision from the service they are replacing. Make sure every on-call member has the right PagerDuty and phone settings before going on call. When servers churn in AWS, stop health checks and autoscaling, for example with the Auto Scaling feature to suspend and resume processes, so the EC2 request limit is not used up.

   View case

By root cause4