Incident Library
Applied filters Root causeConfiguration changeComponentCoordination and key-value storeSymptomCluster or node provisioning stallClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024
    Symptoms

    From 16:30 UTC on 5 August 2024, DigitalOcean's internal services could not discover the other services they depended on. Creates, event processing and the management of DigitalOcean products failed in all regions through the Cloud Control Panel and the API, while existing customer resources such as Droplets stayed online.

    Service outageCluster or node provisioning stall
    Trigger

    During the rollout of a new datastore cluster in a new datacenter, a replication configuration was deployed against the datastore behind internal service discovery that gave the new datacenter, which held zero keys, 100% ownership of all keys.

    Root cause
    Configuration change
    Component
    Coordination and key-value storeService discoveryDatabase replication
    Contributing factor
    Excessive blast radius
    Detection

    Service discovery unavailability was first reported internally at 16:35 UTC, five minutes after the rollout of the new datastore cluster began.

    Remediation
    • Reverted the replication configuration and ran it against the region that had become the datastore owner
    • Applied the configuration by hand on the nodes of lagging regions and ran local data repairs
    • Cleaned up failed events and processed pending events that had not timed out
    Action items
    • Perform this kind of datastore extension inside a declared maintenance window, announced on the status page
    • Update the process documentation for this type of deployment, stage by stage
    • Automate the manual steps used in this deployment to reduce human error
    • Evaluate the internal datastore topology and whether to regionalise or layer it
    View case
Check your cluster for this

This case involves Coordination and key-value store and Database replication. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Coordination and key-value storeDatabase replicationService discovery
Contributing factor
Excessive blast radius
Organisation
DigitalOcean
  1. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024