Incident Library
Applied filters ComponentCoordination and key-value storeFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024
    Symptoms

    From 16:30 UTC on 5 August 2024, DigitalOcean's internal services could not discover the other services they depended on. Creates, event processing and the management of DigitalOcean products failed in all regions through the Cloud Control Panel and the API, while existing customer resources such as Droplets stayed online.

    Service outageCluster or node provisioning stall
    Trigger

    During the rollout of a new datastore cluster in a new datacenter, a replication configuration was deployed against the datastore behind internal service discovery that gave the new datacenter, which held zero keys, 100% ownership of all keys.

    Root cause
    Configuration change
    Component
    Coordination and key-value storeService discoveryDatabase replication
    Contributing factor
    Excessive blast radius
    Detection

    Service discovery unavailability was first reported internally at 16:35 UTC, five minutes after the rollout of the new datastore cluster began.

    Remediation
    • Reverted the replication configuration and ran it against the region that had become the datastore owner
    • Applied the configuration by hand on the nodes of lagging regions and ran local data repairs
    • Cleaned up failed events and processed pending events that had not timed out
    Action items
    • Perform this kind of datastore extension inside a declared maintenance window, announced on the status page
    • Update the process documentation for this type of deployment, stage by stage
    • Automate the manual steps used in this deployment to reduce human error
    • Evaluate the internal datastore topology and whether to regionalise or layer it
    View case
  2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018
    Symptoms

    On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

    High error rateFailover failureMonitoring outage
    Trigger

    A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

    Root cause
    Cloud provider outage
    Component
    Message brokerCoordination and key-value store
    Contributing factor
    Single point of failureExcessive blast radiusMonitoring blind spot
    Remediation
    • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
    • Restarted every Kafka broker in a rolling restart until all partitions were back online
    • Worked through the backlog, including the backlog of alerts, and resumed normal operations
    Action items
    • Isolate the Kafka clusters better across availability zones and regions
    • Stop sharing one ZooKeeper ensemble across Kafka clusters
    • Run game days on this failure and check whether newer Kafka versions fix it
    • Put in place more checks that notify the team earlier about subsystems that need extra help to recover
    Lessons

    Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

    View case
Check your cluster for this

These 2 cases involve Coordination and key-value store and Database replication. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Configuration change1 case

Component
Coordination and key-value storeDatabase replicationService discovery
Contributing factor
Excessive blast radius
Organisation
DigitalOcean
  1. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024

Cloud provider outage1 case

Component
Coordination and key-value storeMessage broker
Contributing factor
Excessive blast radiusMonitoring blind spotSingle point of failure
Organisation
AppOptics
  1. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018