Database backups and snapshots

Cases, by severity

1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hoursPreset · 2026

   Root cause Capacity exhaustion

2. IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026

   Root cause Software defect

3. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeksInfluxData · 2026

   Root cause Faulty deployment

4. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it downMIT CSAIL · 2025

   Root cause Breaking change in an upgrade

5. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hoursMatrix.org Foundation · 2025

   Root cause Operator error

6. IL-0371 Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ranRender · 2023

   Root cause Faulty deployment

7. IL-0453 Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forklessHyperconnect · 2022

   Root cause Capacity exhaustion

8. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updatesBattleMetrics · 2022

   Root cause Data loss or corruption

9. IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicasRazorpay · 2020

   Root cause Configuration change

10. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hoursGitHub · 2018

    Root cause DNS or network fault

11. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deletedDigitalOcean · 2017

    Root cause Configuration change

12. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went downInstapaper · 2017

    Root cause Capacity exhaustion

13. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017

    Root cause Operator error

14. IL-0573 A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stoppedWoowa Brothers · 2016

    Root cause Operator error

15. IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-masterOregon State University Open Source Lab · 2015

    Root cause Operator error

16. IL-0485 Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slowDropbox · 2014

    Root cause Software defect

17. IL-0538 Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backupSakura Internet · 2012

    Root cause Operator error