Cases1 case

1 to 1 of 1

1. IL-0573 A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stopped

   Woowa Brothers2016

   Symptoms

   At 15:00 on 27 December 2016, while a developer was working on a payment system module, nine main tables of the billing database were dropped. The monitoring system turned red, failure alerts poured in, and payments in 배달의민족 (Baemin) stopped for a while.

   Data lossService outage

   Trigger

   While developing a payment system module, the developer who wrote the post dropped nine main tables of the billing database by mistake.

   Root cause

   Operator error

   Component

   DatabaseDatabase backups and snapshots

   Contributing factor

   –

   Detection

   The monitoring system turned red and failure alerts fired.

   Remediation

   • Pointed the application at a temporary database, which kept the outage short
   • Restored the dropped tables from the 6:00 mysqldump full backup
   • Replayed the binary log from 6:00 to 15:00 with mysqlbinlog, filtered to the nine tables and stopping before the DROP TABLE

   Action items

   • Replace mysqldump backups of the billing database with full backups taken by XtraBackup

   Lessons

   There are many points where such failures can be managed: separate database accounts for the service and for administration, check once more before running a query, and guard against rm -rf. Without the full backup and the binary log the data could not have been recovered, so every team should check that it has a full recovery point for a failure.

   View case

By root cause1