Cases1 case

1 to 1 of 1

1. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted

   DigitalOcean2017

   Symptoms

   At 10:24 AM EDT on April 5th, 2017, DigitalOcean began to receive alerts that its public services were not working, and by 10:27 the team had verified that the production database on the primary had been deleted. The control panel and the API were unavailable for four hours and fifty-six minutes: running Droplets kept working, but no new Droplets or other resources could be created or managed.

   Service outage

   Trigger

   A process performing automated testing had been misconfigured with production credentials, and the production primary database was deleted.

   Root cause

   Configuration change

   Component

   DatabaseDatabase replicationDatabase backups and snapshots

   Contributing factor

   No safeguard on destructive change

   Detection

   Alerts at 10:24 AM EDT that public services were not functioning; within three minutes the team found that the primary database had been deleted.

   Remediation

   • Began recovery from a time-delayed database replica
   • Took a backup of the time-delayed replica
   • Copied the backup to the primary and recovered it
   • Copied the backups to the replicas until all systems were restored

   Action items

   • Drastically reduce access to the primary system for certain actions
   • Upgrade network connectivity between database servers
   • Update hardware to speed up recovery
   • Assess further safeguards against developer error
   • Improve the processes around data recovery
   • Explore ways to provide better real-time information to customers during future customer-impacting events

   View case

By root cause1