Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Resources deleted

리소스 삭제 · resources-deleted

13 cases · 2019 to 2026

Root cause Configuration change7 Operator error3 Software defect2 Faulty deployment1

Component Argo CD3 Configuration templating3 Ingress controller3 Terraform3

Contributing factor No safeguard on destructive change8 Automation amplification5 Excessive blast radius5 No runbook4

Organisation 2i2c1 Aqua Security1 Argo CD users1 CircleCI1 DFDS1

Cases

IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSyncArgo CD users · 2026
Root cause Software defect
IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025
Root cause Operator error
IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services

View on the graph

Cloud load balancer2

Kubernetes API server2

Persistent volumes and stateful sets2

Container runtime1

Credential broker for pods1

Application database1

GKE control plane1

Jobs and CronJobs1

Node launch template1

Model request routing1

RBAC and service accounts1

Container registry and its credentials1

Telemetry agent1

Unverified assumption3

Monitoring blind spot3

Missed in review3

Harmful manual action2

No configuration tests2

No staged rollout2

Untested backup1

Unreviewed defaults1

Unrepresentative staging1

UK Ministry of Justice1

Grafana Labs · 2025

Root cause Configuration change

IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024

Root cause Configuration change

IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSetAqua Security · 2023

Root cause Configuration change

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

Root cause Configuration change

IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022

Root cause Operator error

IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022

Root cause Configuration change

IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutesGoogle Cloud · 2022

Root cause Software defect

IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021

Root cause Configuration change

IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.comCircleCI · 2021

Root cause Faulty deployment

IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dnsUK Ministry of Justice · 2020

Root cause Operator error

IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019

Root cause Configuration change