Incident Library
Applied filters Root causeConfiguration changeComponentetcdFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
    Symptoms

    At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

    Resources deletedControl plane unresponsive
    Trigger

    An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

    Root cause
    Configuration change
    Component
    etcdKubernetes API server
    Contributing factor
    Automation amplificationNo safeguard on destructive changeExcessive blast radius
    Detection

    Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

    Remediation
    • restored each affected cluster's etcd from its latest twice-daily backup
    • restored the stateless control plane components after etcd
    • patched the recovery mechanism after unexpected problems during recovery
    • found and fixed an additional small bug during recovery
    • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources
    Action items
    • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment
    Lessons

    When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

    View case
Check your cluster for this

This case involves etcd and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
etcdKubernetes API server
Contributing factor
Automation amplificationExcessive blast radiusNo safeguard on destructive change
Organisation
DigitalOcean
  1. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024