IL-0573 Woowa Brothers · 2016-12-27database
A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stopped
The dropped tables belonged to the MariaDB billing database of Woowa Brothers. The outage stayed short because the application was changed to point at a temporary database. The billing database was backed up in full with mysqldump every day at 6:00, as compressed SQL files per table, so the dropped tables could be restored easily from that backup; with a lot of payment data this took some time, but it brought back the data up to 6:00. The data written between 6:00 and 15:00 was recovered from the MariaDB binary log: mysqlbinlog with --start-datetime="2016-12-27 06:00:00" and --stop-datetime="2016-12-27 15:00:00" turned the binary logs into a restore.log file, which was edited to keep only the nine tables and then applied to the database; the stop time had to be set so that the DROP TABLE statement would not run again. With that, all data was restored.
Later, during a large reorganisation of the billing database onto new servers with partitioning, restoring a single main payment table from a mysqldump backup took an enormous amount of time because order volume and data had grown fast. The team concluded that mysqldump backup and restore was no longer usable, since a similar failure would take tens of hours to recover, and moved the billing database to full physical backups with Percona XtraBackup 2.4 on MariaDB 10.2.x and CentOS 6.x, without incremental backups to keep recovery simple, using the binary log position recorded at backup time for point-in-time recovery.
- Severity
- –
- Scope
- 배달의민족 (Baemin) payments stopped briefly; nine main tables of the billing database were lost until they were restored
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
At 15:00 on 27 December 2016, while a developer was working on a payment system module, nine main tables of the billing database were dropped. The monitoring system turned red, failure alerts poured in, and payments in 배달의민족 (Baemin) stopped for a while.
Symptom Data lossService outage
Root cause
Root cause Operator error
- Trigger
- While developing a payment system module, the developer who wrote the post dropped nine main tables of the billing database by mistake.
Detection
The monitoring system turned red and failure alerts fired.
Remediation
- Pointed the application at a temporary database, which kept the outage short
- Restored the dropped tables from the 6:00 mysqldump full backup
- Replayed the binary log from 6:00 to 15:00 with mysqlbinlog, filtered to the nine tables and stopping before the DROP TABLE
Action items
- Replace mysqldump backups of the billing database with full backups taken by XtraBackup
Lessons
There are many points where such failures can be managed: separate database accounts for the service and for administration, check once more before running a query, and guard against rm -rf. Without the full backup and the binary log the data could not have been recovered, so every team should check that it has a full recovery point for a failure.
Sources
Related cases
IL-0242Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hoursMatrix.org Foundation · 2025same root cause · shared Database backups and snapshots, Database
IL-0218GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017same root cause · shared Database backups and snapshots, Database
IL-0319Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-masterOregon State University Open Source Lab · 2015same root cause · shared Database backups and snapshots, Database
IL-0577Ficilcom's production Neon PostgreSQL database was reset by prisma migrate reset meant for the local databaseFicilcom · 2026same root cause · shared Database
IL-0576Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore themCareNet · 2025same root cause · shared Database
IL-0582Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growingKanmu · 2024same root cause · shared Database
IL-0556A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satelliteStorj · 2024same root cause · shared Database
IL-0531Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of serversMicrosoft Azure · 2024same root cause · shared Database
IL-0565Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outageDevsisters · 2022same root cause · shared Database
IL-0207A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022same root cause · shared Database
IL-0332A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hoursTravis CI · 2018same root cause · shared Database
IL-0431During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017same root cause · shared Database
IL-0506UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwrittenUC Berkeley · 2017same root cause · shared Database
IL-0538Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backupSakura Internet · 2012same root cause · shared Database backups and snapshots
IL-0337EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicasAmazon Web Services · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0573
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0573/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.