Incident Library
Applied filters ComponentDatabase backups and snapshotsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases17 cases

16 to 17 of 17

  1. IL-0485 Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slow
    Dropbox2014
    Symptoms

    On Friday evening, during scheduled maintenance that started at 5:30 PM PT, the Dropbox service went down. Most functionality came back about three hours later, but core service was not fully restored until 4:40 PM PT on Sunday.

    Service outage
    Trigger

    A planned maintenance on Friday at 5:30 PM PT upgraded the operating system on some machines with a script that was meant to check that a machine held no active data before reinstalling it.

    Root cause
    Software defect
    Component
    DatabaseDatabase backups and snapshots
    Contributing factor
    No safeguard on destructive change
    Remediation
    • Restored the affected databases from backups
    Action items
    • Add a check that makes each machine verify its own state locally before running incoming commands, so machines running critical processes refuse destructive operations
    • Use a new tool that replays binary logs in parallel to recover large MySQL backups faster
    • Open source the parallel binary log replay tool
    Lessons

    Scripts that verify a machine's production state remotely before upgrading or repurposing it can be wrong, so machines need to check their own state before accepting destructive commands. Replicas give redundancy, but when every replica is lost the restore from backup decides the length of the outage, and the standard MySQL restore tool is slow on large data sets.

    View case
  2. IL-0538 Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backup
    Sakura Internet2012
    Symptoms

    During emergency maintenance on 5 January 2012, a bulk deletion of unneeded file systems on the Sakura Cloud storage device also deleted 53 disks that customers were using. The deleted disks could not be restored from backup.

    Data loss
    Trigger

    In the emergency maintenance on 5 January 2012, Sakura Internet ran a batch command written only for that maintenance to delete unneeded file systems on the storage device in bulk.

    Root cause
    Operator error
    Component
    Cloud block storageDatabase backups and snapshots
    Contributing factor
    Missed in reviewUntested backup
    Action items
    • Simplify the process for deleting unneeded file systems
    • Do not run scripts that have not been reviewed and tested enough
    View case
Check your cluster for this

These 17 cases involve Database backups and snapshots and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause8

Capacity exhaustion3 cases

Component
Database backups and snapshots3Database2In-memory cache1Database replication1
Contributing factor
Unknown limit2Limited debug access2Unverified assumption1Untested backup1Monitoring blind spot1No alert on limit1No runbook1Unrepresentative staging1
Organisation
Hyperconnect1Instapaper1Preset1
  1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours
    Preset2026
  2. IL-0453 Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forkless
    Hyperconnect2022
  3. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017

Software defect2 cases

Component
Database2Database backups and snapshots2Database replication1Object storage1
Contributing factor
Untested backup1Monitoring blind spot1No safeguard on destructive change1Single point of failure1
Organisation
Dropbox1PostHog1
  1. IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 days
    PostHog2026
  2. IL-0485 Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slow
    Dropbox2014

Faulty deployment2 cases

Component
Database2Database backups and snapshots2Service discovery1
Contributing factor
Missed in review1Unrepresentative staging1Gap in automated tests1Unsafe failure behaviour1
Organisation
InfluxData1Render1
  1. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks
    InfluxData2026
  2. IL-0371 Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ran
    Render2023

Breaking change in an upgrade1 case

Component
Database backups and snapshotsSlurm job scheduler
Contributing factor
Untested backupIneffective rollback
Organisation
MIT CSAIL
  1. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it down
    MIT CSAIL2025

Operator error5 cases

Component
Database backups and snapshots5Database4Database replication3Cloud block storage2
Contributing factor
Untested backup3Unverified assumption2Unreviewed defaults1Unpatched known bug1Harmful manual action1No runbook1No safeguard on destructive change1Overlapping failures1Pages not delivered1Missed in review1Single point of failure1
Organisation
GitLab1Matrix.org Foundation1Oregon State University Open Source Lab1Sakura Internet1Woowa Brothers1
  1. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours
    Matrix.org Foundation2025
  2. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hours
    GitLab2017
  3. IL-0573 A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stopped
    Woowa Brothers2016
  4. IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-master
    Oregon State University Open Source Lab2015
  5. IL-0538 Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backup
    Sakura Internet2012

Data loss or corruption1 case

Component
PostgreSQL vacuumDatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
Monitoring blind spotNo headroomOverlapping failures
Organisation
BattleMetrics
  1. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updates
    BattleMetrics2022

Configuration change2 cases

Component
Database2Database backups and snapshots2Database replication2
Contributing factor
No safeguard on destructive change1
Organisation
DigitalOcean1Razorpay1
  1. IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicas
    Razorpay2020
  2. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted
    DigitalOcean2017

DNS or network fault1 case

Component
DatabaseDatabase backups and snapshotsDatabase replicationDatabase failover managerNIC, cable and switch port link
Contributing factor
Unverified assumptionAutomation amplification
Organisation
GitHub
  1. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hours
    GitHub2018