Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0538 Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backup
    Sakura Internet2012
    Symptoms

    During emergency maintenance on 5 January 2012, a bulk deletion of unneeded file systems on the Sakura Cloud storage device also deleted 53 disks that customers were using. The deleted disks could not be restored from backup.

    Data loss
    Trigger

    In the emergency maintenance on 5 January 2012, Sakura Internet ran a batch command written only for that maintenance to delete unneeded file systems on the storage device in bulk.

    Root cause
    Operator error
    Component
    Cloud block storageDatabase backups and snapshots
    Contributing factor
    Missed in reviewUntested backup
    Action items
    • Simplify the process for deleting unneeded file systems
    • Do not run scripts that have not been reviewed and tested enough
    View case
Check your cluster for this

These 6 cases involve Database backups and snapshots and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Software defect1 case

Component
DatabaseDatabase backups and snapshotsDatabase replicationObject storage
Contributing factor
Untested backupMonitoring blind spotSingle point of failure
Organisation
PostHog
  1. IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 days
    PostHog2026

Breaking change in an upgrade1 case

Component
Database backups and snapshotsSlurm job scheduler
Contributing factor
Untested backupIneffective rollback
Organisation
MIT CSAIL
  1. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it down
    MIT CSAIL2025

Capacity exhaustion1 case

Component
DatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
Untested backupUnknown limitLimited debug accessMonitoring blind spotNo runbook
Organisation
Instapaper
  1. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017

Operator error3 cases

Component
Database backups and snapshots3Cloud block storage2Database2Database replication2
Contributing factor
Untested backup3Unverified assumption2Unreviewed defaults1No runbook1Pages not delivered1Missed in review1Single point of failure1
Organisation
GitLab1Oregon State University Open Source Lab1Sakura Internet1
  1. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hours
    GitLab2017
  2. IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-master
    Oregon State University Open Source Lab2015
  3. IL-0538 Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backup
    Sakura Internet2012