IL-0374 DigitalOcean · 2024-08-05database
DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
DigitalOcean was adding a new datacenter to the datastore that powers its internal service discovery. The replication configuration deployed for it named the new datacenter, which held zero keys, as the owner of 100% of the keys in the datastore. The configuration propagated globally at once, the datastore nodes lost quorum in every region, and clients could no longer read from or write to the datastore, which cascaded to every service that finds its dependencies through service discovery.
Engineers identified the lack of quorum and datastore ownership as the blocking issue at 16:42 UTC and the replication change as its cause at 17:00 UTC, then reverted the configuration at 17:16 UTC and ran it against the region that had become the datastore owner. Some regions picked up the corrected configuration and recovered. For the regions that had not, engineers applied the configuration directly on the nodes from 18:25 UTC and ran local repairs on the data to make sure it was aligned before moving on to the next region. After failed events were cleaned up and pending events that had not yet timed out were processed, the platform stabilised at 19:10 UTC. The replication configuration deployment had taken place outside a normal maintenance window.
- Severity
- –
- Scope
- Creates, event processing and management through the Cloud Control Panel and the API for Droplets, networking, login and authentication, Block Storage snapshots, Spaces and CDN, Managed Kubernetes and Managed Databases; existing resources stayed available
- Users affected
- –
- Region
- Global
- Time to detect
- 5m
- Time to resolve
- 2h 40m
Symptoms
From 16:30 UTC on 5 August 2024, DigitalOcean's internal services could not discover the other services they depended on. Creates, event processing and the management of DigitalOcean products failed in all regions through the Cloud Control Panel and the API, while existing customer resources such as Droplets stayed online.
The first visible effect was a complete disruption of reads and writes to the datastore behind internal service discovery, whose nodes lost quorum in every region. Droplet creates and updates, network creates, login and authentication, Block Storage volume snapshot creation, Spaces and CDN creates, Spaces updates, Managed Kubernetes cluster creates and Managed Databases creates failed. Other services also rely on service discovery to talk to each other, so customers may have seen further impact when managing them through the Cloud Control Panel or the API.
Events failed with a distinct timeout at first. After the replication configuration was reverted at 17:16 UTC, some events failed faster, and the error changed from a timeout to a failure to find endpoints. Some regions took longer to pick up the reverted configuration and kept failing until engineers applied it on their nodes by hand.
Root cause
Root cause Configuration change
- Trigger
- During the rollout of a new datastore cluster in a new datacenter, a replication configuration was deployed against the datastore behind internal service discovery that gave the new datacenter, which held zero keys, 100% ownership of all keys.
Detection
Service discovery unavailability was first reported internally at 16:35 UTC, five minutes after the rollout of the new datastore cluster began.
Remediation
- Reverted the replication configuration and ran it against the region that had become the datastore owner
- Applied the configuration by hand on the nodes of lagging regions and ran local data repairs
- Cleaned up failed events and processed pending events that had not timed out
Action items
- Perform this kind of datastore extension inside a declared maintenance window, announced on the status page
- Update the process documentation for this type of deployment, stage by stage
- Automate the manual steps used in this deployment to reduce human error
- Evaluate the internal datastore topology and whether to regionalise or layer it
Lessons
–
Sources
- https://status.digitalocean.com/incidents/222822dwq709 · official-status · archived copy
Related cases
IL-0560Some PubNub messages in the US East point of presence were misrouted with no error and not delivered right awayPubNub · 2026same root cause · shared Service discovery
IL-0499Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at onceDiscord · 2026same root cause · shared Coordination and key-value store
IL-0346Render's routing service returned intermittent 404s for web services and static sites in OregonRender · 2025same root cause · shared Service discovery
IL-0392MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook runWikimedia Foundation · 2024same root cause · shared Database replication
IL-0191A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022same root cause · shared Service discovery
IL-0331An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicasRazorpay · 2020same root cause · shared Database replication
IL-0430Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degradedHatena · 2018same root cause · shared Database replication
IL-0325DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deletedDigitalOcean · 2017same root cause · shared Database replication
IL-0215Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedlyTwilio · 2013same root cause · shared Database replication
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0374
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0374/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.