Cases11 cases

1 to 5 of 11

1. IL-0577 Ficilcom's production Neon PostgreSQL database was reset by prisma migrate reset meant for the local database

   Ficilcom2026

   Symptoms

   On the first working day after the Silver Week holidays, an engineer ran prisma migrate reset and then loaded seed data, intending to clean the local database before resuming work. The seed data did not seem to be there, and the engineer first suspected that the coding agent had skipped the step. The reset had in fact run against the production database.

   Data loss

   Trigger

   The engineer ran prisma migrate reset believing it targeted the local database, while packages/database/.env still held the production DATABASE_URL left there from a production migration before the holidays.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   No safeguard on destructive change

   Detection

   After the reset the engineer saw that the seed data did not seem to be present and realised the reset had hit production.

   Remediation

   • Restored the production database with Neon's Backup & Restore

   Action items

   • Make local scripts check DATABASE_URL and refuse to run destructive commands against anything but localhost or 127.0.0.1
   • Stop writing production connection strings in local .env files
   • Run production operations through CI or a secret manager
   • Check local env files for production hosts such as *.neon.tech
   • Check the Backup & Restore history window of the plan in use, and extend it or add separate backups if it is not enough
   • Before holidays or long breaks, remove the production URL from local env files
   • When stopping work midway, record how far it got in an issue or memo
   • Add rules that forbid coding agents from running destructive database commands unless the user asks in the current message

   Lessons

   Memory after a long holiday cannot be trusted, so the state of unfinished work has to be written down instead. A destructive command should refuse to run when its target is not clearly local. That a backup feature exists does not mean the plan's history window is long enough, and the damage was small this time only because the service happened to be at an early stage.

   View case
2. IL-0576 Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore them

   CareNet2025

   Symptoms

   On a day in August 2025, at 10:41, CareNet's system development division posted a first report on Slack that a failure was occurring in BigQuery, the production data warehouse. At 11:47 the division's investigation found that the production datasets had been deleted by a service account belonging to the AI technology development team.

   Data loss

   Trigger

   Test code that was being modified contained a step that resets test data by deleting a dataset and recreating it with the same name. Because of a configuration mistake, the step ran against production and deleted and recreated every production dataset.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   No safeguard on destructive change

   Detection

   The system development division posted the first report of a BigQuery failure on Slack at 10:41, and its investigation found at 11:47 that the AI technology development team's service account had deleted the production datasets.

   Remediation

   • Restored the tables one at a time with bq cp in Cloud Shell from BigQuery time travel, to a point before the incident
   • Recreated views and table functions, which bq cp cannot restore, from the creation queries kept in BigQuery saved queries

   Action items

   • Enforce controls so that test code cannot access the production environment
   • Remove the excessive permissions granted to the service account
   • Keep dataset and table information as managed metadata, because restoring with bq cp requires knowing the table names
   • Keep the creation queries of views and table functions in a place such as GitHub, because bq cp cannot restore them
   • Consider how BigQuery should be backed up

   Lessons

   BigQuery time travel with bq cp recovered most of the data, but only because the table names were known, and it does not bring back views or table functions. Test code should not be able to reach production, and service accounts should not hold more permissions than they need.

   View case
3. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours

   Matrix.org Foundation2025

   Symptoms

   On 2 September 2025 the matrix.org homeserver went down at 17:45 UTC and stayed down until 18:00 UTC the next day. Both machines of its PostgreSQL cluster were out of action: the original primary had lost its RAID array during a disk expansion, and the data directory of the new primary had been deleted by mistake.

   Service outageData loss

   Trigger

   During maintenance to add drives, an existing drive disappeared from the primary database server's RAID 10 array at 11:17 UTC, which led to a failover and a rebuild of that server. The outage itself began when the data directory of the new primary on db-01 was deleted while clearing a failed restore on db-02.

   Root cause

   Operator error

   Component

   DatabaseDatabase backups and snapshotsDatabase replication

   Contributing factor

   Overlapping failuresHarmful manual actionUnpatched known bugNo safeguard on destructive change

   Detection

   Monitoring fired when the drive disappeared from db-02's RAID array at 11:17 UTC, and Mythic Beasts confirmed the issue. The team realised it had deleted db-01's data directory, and the backend team was then paged, an incident was opened and an emergency was declared.

   Remediation

   • Created a fresh RAID 10 array on db-02
   • Unblocked WAL archiving from the new primary on db-01 to S3
   • Started restoring db-02 from the S3 backups with wal-g
   • Shut down Synapse and the damaged primary database
   • Remounted the data partition of db-01 read-only
   • Copied the WALs missing from S3 from db-01 to db-02
   • Updated wal-g and patched it to continue from the partial restore
   • Configured a restore_command wrapper that fetched WALs from S3 and the local files, and replayed them in standby mode
   • Promoted db-02 to primary and restarted Synapse
   • Ran backups against the primary while no replica existed
   • Rebuilt db-01 as a replica from a base backup of db-02

   Action items

   • Run pg_repack to reclaim free space
   • Take incremental backups more often, since they restore faster than WAL replay
   • Reduce Synapse's storage requirements
   • Consider changing the terminal background colour by the database's role, for example by watching for the standby.signal file
   • Consider wrapper scripts around sensitive commands, such as an alias for rm, and automating operations such as starting a base backup from primary to secondary
   • Shorten the recovery time, with infrastructure from service providers
   • Upgrade wal-g on all servers and repeat disaster recovery testing with it
   • Explore ZFS local snapshots at the next hardware refresh
   • Make status.matrix.org the source of truth during incidents and keep social media updated

   Lessons

   Most of the difficulty came from the size of the database: restoring 51TB and replaying WALs took far longer than hoped, and incremental backups restore much faster than WAL replay. Running a destructive command on the wrong server was the turning point, and Matrix.org looks for better tooling and processes rather than blaming the individual; working as a trio did not catch the mistake, and server names that suggest a role caused confusion because a primary and a secondary swap roles. The recovery lost no data, but the recovery time needs to be shorter, and the wal-g versions on the disaster recovery server and the production servers had drifted apart.

   View case
4. IL-0565 Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outage

   Devsisters2022

   Symptoms

   Right after a script ran on the CockroachDB nodes behind Cookie Run: Kingdom, hundreds of alerts fired at once for error rate, database connections and health checks, and the game could no longer be played normally. Emergency maintenance started at 16:52.

   Service outageData lossDatabase overload

   Trigger

   On the Monday after the launch weekend, storage use was growing with the number of users and the cluster had about 36 hours left before its disks would fill. The team ran a prepared script to create ballast files on every node, as the official guide for CockroachDB 19.1.11 recommends. The ballast file path in the script was set wrong and pointed at a block device, and the command had no check for an existing file at the target path, so it overwrote the partition.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   No safeguard on destructive changeOverlapping failuresRetry storm

   Detection

   Metrics and alerts set up in advance fired hundreds of alerts on error rate, database connections and health checks right after the script ran.

   Remediation

   • Started emergency maintenance of the game at 16:52
   • Wrote Spark code that extracts the contents of CockroachDB SST files to CSV, and modified the CockroachDB CLI to make it more than 10 times faster
   • Converted about 7.5TB of SST files from the 7 surviving nodes to CSV
   • Deduplicated the extracted rows and merged them into one large CSV file that could be imported
   • Imported the latest snapshot and the latest 300 journal entries per user into a new cluster with more storage
   • Restored the other tables from the latest backup and rebuilt the user table from snapshots and journals with server logic
   • Checked the restored data against a database restored from backup, against analytics logs for sampled users, and with snapshot replay for every user
   • Reopened the game at 23:37 on the second day, after 30 hours 50 minutes
   • Failed over the AWS RDS database of the DevPlay platform after the reopening traffic overloaded it
   • Built a third database cluster on more m5d instances and moved the data with the official backup and restore
   • Ended the maintenance at 8:30, after 36 hours 29 minutes

   Action items

   • Require at least two engineers for any work on production infrastructure, however small
   • Write the scripts and the scenario for each infrastructure task in advance and have the team review them

   Lessons

   Because a person's mistake caused the failure, Devsisters changed its process for infrastructure work rather than relying on individual care: even experienced engineers make mistakes, so the prevention has to be built into the system. The author credits the recovery to an incident policy that does not blame the individual and asks for detailed records, to past incidents that were written down so people without the experience knew what to do, and to having infrastructure operations and data analysis teams with the needed infrastructure and skills in one organisation.

   View case
5. IL-0332 A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hours

   Travis CI2018

   Symptoms

   On 13 March 2018 at 12:14 UTC every table in travis-ci.com's production PostgreSQL database was truncated. Alerts fired right away, and customers reported missing account data; people who signed in saw blank user profiles.

   Data lossService outage

   Trigger

   At 12:04 UTC a developer ran a test suite in an old tmux shell that still had a DATABASEURL environment variable pointing at the production database, and the Database Cleaner gem sent a truncate query to it. The query was blocked for about 10 minutes and executed at 12:14 UTC.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   No safeguard on destructive changeHarmful manual action

   Detection

   Alerts fired immediately after the truncation, and customers reported missing account data.

   Remediation

   • Took all applications offline
   • Provisioned a replacement database from a snapshot taken that morning
   • Reattached the restored database and brought platform services back up
   • Worked through the backlog of builds until it cleared
   • Revoked all affected login tokens by 14:22 UTC on 14 March
   • Contacted affected customers and advised them to rotate stored credentials and check build logs

   Action items

   • Revoke the truncate permission on the databases
   • Make the internal spechelpers check for the DATABASEURL environment variable
   • Make the developer shell prompt warn when a DATABASE_URL is set
   • Submit a pull request to the Database Cleaner gem guarding against a remote DATABASE_URL
   • Create an alias for the follower database so it is easy to connect to for work against production data
   • Automate database failover and maintenance to cut manual recovery steps

   Lessons

   Developers connected to the production primary with write access because the tooling made the read-only follower hard to reach. While responding, Travis CI left user-facing applications running, which created duplicate user ids, and did not turn some alerts back on, which hid the stopped cron scheduler. Even the most experienced developers can make an inadvertent mistake that causes a large outage. The whole production database was recovered with only about 15 minutes of data loss.

   View case

By root cause1