Incident Library
Applied filters ComponentDatabase backups and snapshotsFactorNo safeguard on destructive changeOrganisationDigitalOceanClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted
    DigitalOcean2017
    Symptoms

    At 10:24 AM EDT on April 5th, 2017, DigitalOcean began to receive alerts that its public services were not working, and by 10:27 the team had verified that the production database on the primary had been deleted. The control panel and the API were unavailable for four hours and fifty-six minutes: running Droplets kept working, but no new Droplets or other resources could be created or managed.

    Service outage
    Trigger

    A process performing automated testing had been misconfigured with production credentials, and the production primary database was deleted.

    Root cause
    Configuration change
    Component
    DatabaseDatabase replicationDatabase backups and snapshots
    Contributing factor
    No safeguard on destructive change
    Detection

    Alerts at 10:24 AM EDT that public services were not functioning; within three minutes the team found that the primary database had been deleted.

    Remediation
    • Began recovery from a time-delayed database replica
    • Took a backup of the time-delayed replica
    • Copied the backup to the primary and recovered it
    • Copied the backups to the replicas until all systems were restored
    Action items
    • Drastically reduce access to the primary system for certain actions
    • Upgrade network connectivity between database servers
    • Update hardware to speed up recovery
    • Assess further safeguards against developer error
    • Improve the processes around data recovery
    • Explore ways to provide better real-time information to customers during future customer-impacting events
    View case
Check your cluster for this

This case involves Database and Database backups and snapshots. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
DatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
No safeguard on destructive change
Organisation
DigitalOcean
  1. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted
    DigitalOcean2017