Cases1 case

1 to 1 of 1

1. IL-0485 Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slow

   Dropbox2014

   Symptoms

   On Friday evening, during scheduled maintenance that started at 5:30 PM PT, the Dropbox service went down. Most functionality came back about three hours later, but core service was not fully restored until 4:40 PM PT on Sunday.

   Service outage

   Trigger

   A planned maintenance on Friday at 5:30 PM PT upgraded the operating system on some machines with a script that was meant to check that a machine held no active data before reinstalling it.

   Root cause

   Software defect

   Component

   DatabaseDatabase backups and snapshots

   Contributing factor

   No safeguard on destructive change

   Remediation

   • Restored the affected databases from backups

   Action items

   • Add a check that makes each machine verify its own state locally before running incoming commands, so machines running critical processes refuse destructive operations
   • Use a new tool that replays binary logs in parallel to recover large MySQL backups faster
   • Open source the parallel binary log replay tool

   Lessons

   Scripts that verify a machine's production state remotely before upgrading or repurposing it can be wrong, so machines need to check their own state before accepting destructive commands. Replicas give redundancy, but when every replica is lost the restore from backup decides the length of the outage, and the standard MySQL restore tool is slow on large data sets.

   View case

By root cause1