Incident Library
Applied filters ComponentKubernetes API serverSymptomResources deletedClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
    Symptoms

    At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

    Resources deletedControl plane unresponsive
    Trigger

    An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

    Root cause
    Configuration change
    Component
    etcdKubernetes API server
    Contributing factor
    Automation amplificationNo safeguard on destructive changeExcessive blast radius
    Detection

    Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

    Remediation
    • restored each affected cluster's etcd from its latest twice-daily backup
    • restored the stateless control plane components after etcd
    • patched the recovery mechanism after unexpected problems during recovery
    • found and fixed an additional small bug during recovery
    • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources
    Action items
    • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment
    Lessons

    When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

    View case
  2. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
    Symptoms

    Aqua's research team Nautilus recorded the attack on one of its Kubernetes honeypots. What appeared in the cluster was a DaemonSet running the image kuberntesio/kube-controller:1.0.1 from Docker Hub on every node, using the nodes' CPU to mine Monero.

    Unauthorised access or a foreign workloadResources deleted
    Trigger

    An API server misconfigured to accept unauthenticated requests from anonymous users with privileges.

    Root cause
    Configuration change
    Component
    Kubernetes API serverRBAC and service accountsContainer registry and its credentials
    Contributing factor
    Monitoring blind spot
    Detection

    Aqua recorded the attacker's API requests on its honeypot; planted AWS access keys raised a beacon when the attacker tried to use them.

    Lessons

    Fixing an exposed API server is not enough once an attacker has been in: a ClusterRoleBinding to an attacker-created ServiceAccount keeps working after anonymous access is closed. Attackers name these objects and images after real Kubernetes components so they blend into the API audit logs and pass for legitimate deployments. Aqua recommends scanning clusters for vulnerabilities, exposed secrets and misconfigurations with a tool such as Trivy.

    View case
Check your cluster for this

These 2 cases involve Kubernetes API server and etcd. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Kubernetes API server2etcd1RBAC and service accounts1Container registry and its credentials1
Contributing factor
Automation amplification1Excessive blast radius1Monitoring blind spot1No safeguard on destructive change1
Organisation
Aqua Security1DigitalOcean1
  1. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  2. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023