Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0538 Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backup
    Sakura Internet2012
    Symptoms

    During emergency maintenance on 5 January 2012, a bulk deletion of unneeded file systems on the Sakura Cloud storage device also deleted 53 disks that customers were using. The deleted disks could not be restored from backup.

    Data loss
    Trigger

    In the emergency maintenance on 5 January 2012, Sakura Internet ran a batch command written only for that maintenance to delete unneeded file systems on the storage device in bulk.

    Root cause
    Operator error
    Component
    Cloud block storageDatabase backups and snapshots
    Contributing factor
    Missed in reviewUntested backup
    Action items
    • Simplify the process for deleting unneeded file systems
    • Do not run scripts that have not been reviewed and tested enough
    View case
Check your cluster for this

These 8 cases involve Database backups and snapshots and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Software defect1 case

Component
DatabaseDatabase backups and snapshotsDatabase replicationObject storage
Contributing factor
Untested backupMonitoring blind spotSingle point of failure
Organisation
PostHog
  1. IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 days
    PostHog2026

Faulty deployment1 case

Component
DatabaseDatabase backups and snapshots
Contributing factor
Missed in reviewUnrepresentative stagingGap in automated testsUnsafe failure behaviour
Organisation
InfluxData
  1. IL-0380 A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeks
    InfluxData2026

Operator error5 cases

Component
Database backups and snapshots5Database4Database replication3Cloud block storage2
Contributing factor
Untested backup3Unverified assumption2Unreviewed defaults1Unpatched known bug1Harmful manual action1No runbook1No safeguard on destructive change1Overlapping failures1Pages not delivered1Missed in review1Single point of failure1
Organisation
GitLab1Matrix.org Foundation1Oregon State University Open Source Lab1Sakura Internet1Woowa Brothers1
  1. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours
    Matrix.org Foundation2025
  2. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hours
    GitLab2017
  3. IL-0573 A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stopped
    Woowa Brothers2016
  4. IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-master
    Oregon State University Open Source Lab2015
  5. IL-0538 Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backup
    Sakura Internet2012

Configuration change1 case

Component
DatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
–
Organisation
Razorpay
  1. IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicas
    Razorpay2020