DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted

Symptoms

At 10:24 AM EDT on April 5th, 2017, DigitalOcean began to receive alerts that its public services were not working, and by 10:27 the team had verified that the production database on the primary had been deleted. The control panel and the API were unavailable for four hours and fifty-six minutes: running Droplets kept working, but no new Droplets or other resources could be created or managed.

Recovery from a time-delayed database replica began at 10:34. A backup of that replica was complete at 11:53, the copy to the primary finished at 12:34, and recovery of the primary was complete at 13:31 while copies to the replicas continued. All systems were restored at 15:20 with no data lost. Most of the time went into copying data between the replicas over the network and restoring it into an active server.

Symptom Service outage

Root cause

Trigger

A process performing automated testing had been misconfigured with production credentials, and the production primary database was deleted.

Detection

Alerts at 10:24 AM EDT that public services were not functioning; within three minutes the team found that the primary database had been deleted.

Remediation

• Began recovery from a time-delayed database replica
• Took a backup of the time-delayed replica
• Copied the backup to the primary and recovered it
• Copied the backups to the replicas until all systems were restored

Action items

• Drastically reduce access to the primary system for certain actions
• Upgrade network connectivity between database servers
• Update hardware to speed up recovery
• Assess further safeguards against developer error
• Improve the processes around data recovery
• Explore ways to provide better real-time information to customers during future customer-impacting events

Lessons

–

Sources

• https://www.digitalocean.com/company/blog/update-on-the-april-5th-2017-outage/ · blog · archived copy

Related cases

• IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicasRazorpay · 2020

  same root cause · shared Database replication, Database, Database backups and snapshots

• IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook runWikimedia Foundation · 2024

  same root cause · shared Database replication, Database

• IL-0574 Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS ProxySafie · 2026

  same root cause · shared Database

• IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backendsGoogle Cloud · 2026

  same root cause · shared Database

• IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026

  same root cause · shared Database

• IL-0561 DigitalOcean's Cloud Control Panel and Public API returned up to 25% errors in two windows as authorization timed outDigitalOcean · 2026

  same root cause · shared Database

• IL-0487 Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexedAutumn · 2026

  same root cause · shared Database

• IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature fileCloudflare · 2025

  same root cause · shared Database

• IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hoursGoogle Cloud · 2025

  same root cause · shared Database

• IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorumDigitalOcean · 2024

  same root cause · shared Database replication

• IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on themMicrosoft Azure · 2024

  same root cause · shared Database

• IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'MIT Open Learning · 2023

  same root cause · shared Database

• IL-0272 New Cassandra nodes served reads for data they did not hold, so Monzo records looked missing and payments failedMonzo · 2019

  same root cause · shared Database

• IL-0430 Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degradedHatena · 2018

  same root cause · shared Database replication

• IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be readGoogle Cloud · 2017

  same root cause · shared Database

• IL-0215 Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedlyTwilio · 2013

  same root cause · shared Database replication

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0325
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0325/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.