Incident Library
Applied filters ComponentDatabase backups and snapshotsSymptomService outageFactorNo safeguard on destructive changeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours
    Matrix.org Foundation2025
    Symptoms

    On 2 September 2025 the matrix.org homeserver went down at 17:45 UTC and stayed down until 18:00 UTC the next day. Both machines of its PostgreSQL cluster were out of action: the original primary had lost its RAID array during a disk expansion, and the data directory of the new primary had been deleted by mistake.

    Service outageData loss
    Trigger

    During maintenance to add drives, an existing drive disappeared from the primary database server's RAID 10 array at 11:17 UTC, which led to a failover and a rebuild of that server. The outage itself began when the data directory of the new primary on db-01 was deleted while clearing a failed restore on db-02.

    Root cause
    Operator error
    Component
    DatabaseDatabase backups and snapshotsDatabase replication
    Contributing factor
    Overlapping failuresHarmful manual actionUnpatched known bugNo safeguard on destructive change
    Detection

    Monitoring fired when the drive disappeared from db-02's RAID array at 11:17 UTC, and Mythic Beasts confirmed the issue. The team realised it had deleted db-01's data directory, and the backend team was then paged, an incident was opened and an emergency was declared.

    Remediation
    • Created a fresh RAID 10 array on db-02
    • Unblocked WAL archiving from the new primary on db-01 to S3
    • Started restoring db-02 from the S3 backups with wal-g
    • Shut down Synapse and the damaged primary database
    • Remounted the data partition of db-01 read-only
    • Copied the WALs missing from S3 from db-01 to db-02
    • Updated wal-g and patched it to continue from the partial restore
    • Configured a restore_command wrapper that fetched WALs from S3 and the local files, and replayed them in standby mode
    • Promoted db-02 to primary and restarted Synapse
    • Ran backups against the primary while no replica existed
    • Rebuilt db-01 as a replica from a base backup of db-02
    Action items
    • Run pg_repack to reclaim free space
    • Take incremental backups more often, since they restore faster than WAL replay
    • Reduce Synapse's storage requirements
    • Consider changing the terminal background colour by the database's role, for example by watching for the standby.signal file
    • Consider wrapper scripts around sensitive commands, such as an alias for rm, and automating operations such as starting a base backup from primary to secondary
    • Shorten the recovery time, with infrastructure from service providers
    • Upgrade wal-g on all servers and repeat disaster recovery testing with it
    • Explore ZFS local snapshots at the next hardware refresh
    • Make status.matrix.org the source of truth during incidents and keep social media updated
    Lessons

    Most of the difficulty came from the size of the database: restoring 51TB and replaying WALs took far longer than hoped, and incremental backups restore much faster than WAL replay. Running a destructive command on the wrong server was the turning point, and Matrix.org looks for better tooling and processes rather than blaming the individual; working as a trio did not catch the mistake, and server names that suggest a role caused confusion because a primary and a secondary swap roles. The recovery lost no data, but the recovery time needs to be shorter, and the wal-g versions on the disaster recovery server and the production servers had drifted apart.

    View case
  2. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted
    DigitalOcean2017
    Symptoms

    At 10:24 AM EDT on April 5th, 2017, DigitalOcean began to receive alerts that its public services were not working, and by 10:27 the team had verified that the production database on the primary had been deleted. The control panel and the API were unavailable for four hours and fifty-six minutes: running Droplets kept working, but no new Droplets or other resources could be created or managed.

    Service outage
    Trigger

    A process performing automated testing had been misconfigured with production credentials, and the production primary database was deleted.

    Root cause
    Configuration change
    Component
    DatabaseDatabase replicationDatabase backups and snapshots
    Contributing factor
    No safeguard on destructive change
    Detection

    Alerts at 10:24 AM EDT that public services were not functioning; within three minutes the team found that the primary database had been deleted.

    Remediation
    • Began recovery from a time-delayed database replica
    • Took a backup of the time-delayed replica
    • Copied the backup to the primary and recovered it
    • Copied the backups to the replicas until all systems were restored
    Action items
    • Drastically reduce access to the primary system for certain actions
    • Upgrade network connectivity between database servers
    • Update hardware to speed up recovery
    • Assess further safeguards against developer error
    • Improve the processes around data recovery
    • Explore ways to provide better real-time information to customers during future customer-impacting events
    View case
  3. IL-0485 Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slow
    Dropbox2014
    Symptoms

    On Friday evening, during scheduled maintenance that started at 5:30 PM PT, the Dropbox service went down. Most functionality came back about three hours later, but core service was not fully restored until 4:40 PM PT on Sunday.

    Service outage
    Trigger

    A planned maintenance on Friday at 5:30 PM PT upgraded the operating system on some machines with a script that was meant to check that a machine held no active data before reinstalling it.

    Root cause
    Software defect
    Component
    DatabaseDatabase backups and snapshots
    Contributing factor
    No safeguard on destructive change
    Remediation
    • Restored the affected databases from backups
    Action items
    • Add a check that makes each machine verify its own state locally before running incoming commands, so machines running critical processes refuse destructive operations
    • Use a new tool that replays binary logs in parallel to recover large MySQL backups faster
    • Open source the parallel binary log replay tool
    Lessons

    Scripts that verify a machine's production state remotely before upgrading or repurposing it can be wrong, so machines need to check their own state before accepting destructive commands. Replicas give redundancy, but when every replica is lost the restore from backup decides the length of the outage, and the standard MySQL restore tool is slow on large data sets.

    View case
Check your cluster for this

These 3 cases involve Database and Database backups and snapshots. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Operator error1 case

Component
DatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
Unpatched known bugHarmful manual actionNo safeguard on destructive changeOverlapping failures
Organisation
Matrix.org Foundation
  1. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours
    Matrix.org Foundation2025

Configuration change1 case

Component
DatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
No safeguard on destructive change
Organisation
DigitalOcean
  1. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted
    DigitalOcean2017

Software defect1 case

Component
DatabaseDatabase backups and snapshots
Contributing factor
No safeguard on destructive change
Organisation
Dropbox
  1. IL-0485 Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slow
    Dropbox2014