Cases2 cases

1 to 2 of 2

1. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum

   DigitalOcean2024

   Symptoms

   From 16:30 UTC on 5 August 2024, DigitalOcean's internal services could not discover the other services they depended on. Creates, event processing and the management of DigitalOcean products failed in all regions through the Cloud Control Panel and the API, while existing customer resources such as Droplets stayed online.

   Service outageCluster or node provisioning stall

   Trigger

   During the rollout of a new datastore cluster in a new datacenter, a replication configuration was deployed against the datastore behind internal service discovery that gave the new datacenter, which held zero keys, 100% ownership of all keys.

   Root cause

   Configuration change

   Component

   Coordination and key-value storeService discoveryDatabase replication

   Contributing factor

   Excessive blast radius

   Detection

   Service discovery unavailability was first reported internally at 16:35 UTC, five minutes after the rollout of the new datastore cluster began.

   Remediation

   • Reverted the replication configuration and ran it against the region that had become the datastore owner
   • Applied the configuration by hand on the nodes of lagging regions and ran local data repairs
   • Cleaned up failed events and processed pending events that had not timed out

   Action items

   • Perform this kind of datastore extension inside a declared maintenance window, announced on the status page
   • Update the process documentation for this type of deployment, stage by stage
   • Automate the manual steps used in this deployment to reduce human error
   • Evaluate the internal datastore topology and whether to regionalise or layer it

   View case
2. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted

   DigitalOcean2017

   Symptoms

   At 10:24 AM EDT on April 5th, 2017, DigitalOcean began to receive alerts that its public services were not working, and by 10:27 the team had verified that the production database on the primary had been deleted. The control panel and the API were unavailable for four hours and fifty-six minutes: running Droplets kept working, but no new Droplets or other resources could be created or managed.

   Service outage

   Trigger

   A process performing automated testing had been misconfigured with production credentials, and the production primary database was deleted.

   Root cause

   Configuration change

   Component

   DatabaseDatabase replicationDatabase backups and snapshots

   Contributing factor

   No safeguard on destructive change

   Detection

   Alerts at 10:24 AM EDT that public services were not functioning; within three minutes the team found that the primary database had been deleted.

   Remediation

   • Began recovery from a time-delayed database replica
   • Took a backup of the time-delayed replica
   • Copied the backup to the primary and recovered it
   • Copied the backups to the replicas until all systems were restored

   Action items

   • Drastically reduce access to the primary system for certain actions
   • Upgrade network connectivity between database servers
   • Update hardware to speed up recovery
   • Assess further safeguards against developer error
   • Improve the processes around data recovery
   • Explore ways to provide better real-time information to customers during future customer-impacting events

   View case

By root cause1