Incident Library
Applied filters SymptomResources deletedClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases13 cases

12 to 13 of 13

  1. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
    Symptoms

    At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

    Service outageResources deletedMonitoring outageStale endpoint routing
    Trigger

    A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

    Root cause
    Operator error
    Component
    Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus
    Contributing factor
    No safeguard on destructive change
    Detection

    The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

    Remediation
    • restored the default ingress controller together with external-dns and kiam, which it depends on
    • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
    • restored the remaining components with a targeted apply
    • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository
    View case
  2. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
    Symptoms

    Terraform, applied from the shared cluster configuration repository after a merge, deleted Spotify's production GKE cluster in Asia. While the team tried to recreate it, Terraform deleted a second production cluster, so two of the three production clusters were gone.

    Resources deleted
    Trigger

    A feature team's pull request that added three users to an existing RBAC role binding file was merged before the cluster operators' pull request that imported the Asia cluster into Terraform state. The review build of the import pull request had already written the cluster into the shared state file, and the merged configuration had no definition of the Asia cluster, so Terraform removed it.

    Root cause
    Configuration change
    Component
    TerraformCI systemGKE control plane
    Contributing factor
    No safeguard on destructive changeMissed in reviewUnverified assumption
    Detection

    Internal service discovery could not reach the deleted cluster and the cluster operator team was paged.

    Remediation
    • removed the deleted clusters from the service discovery configuration and restarted service discovery so clients fell back to the instances outside Kubernetes
    • one team that ran half on Kubernetes and half on VMs created more VMs outside Kubernetes to handle the load
    • recreated the clusters, using the hourly backups
    • updated the places where the old clusters' master IPs were hardcoded
    Action items
    • Post the Terraform dry-run output as a comment on each pull request
    • Require a status check that the branch is up to date with master, so the dry run reflects what will actually happen
    • Require an approving review on pull requests
    • Fail reviews automatically when certain keywords such as destroy appear; cluster deletions are done manually outside this repository
    • Add linters and validators for the cluster configuration
    • Schedule and announce disaster recovery tests that cover a cluster going away and smaller failures
    Lessons

    Plan for failure from the start and migrate large infrastructure gradually, so there is redundancy and a way back. A backup whose restore has never been tested is not a backup. Practising disaster recovery paid off: restoring the first deleted cluster took three and a quarter hours, and the team can now restore a larger cluster in one hour. What matters is not who made the mistake but what was learned from it.

    View case
Check your cluster for this

These 13 cases involve Argo CD and Configuration templating. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Software defect2 cases

Component
Argo CD1Model request routing1
Contributing factor
Unverified assumption1Automation amplification1Excessive blast radius1Harmful manual action1No safeguard on destructive change1
Organisation
Argo CD users1Google Cloud1
  1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync
    Argo CD users2026
  2. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022

Operator error3 cases

Component
Ingress controller2Cloud load balancer1Credential broker for pods1Application database1external-dns1Jobs and CronJobs1Persistent volumes and stateful sets1Prometheus1Terraform1
Contributing factor
No safeguard on destructive change3No runbook1
Organisation
2i2c1GitLab1UK Ministry of Justice1
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
  2. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  3. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020

Configuration change7 cases

Component
Configuration templating3Argo CD2Kubernetes API server2Terraform2CI system1Cloud load balancer1Container runtime1etcd1GKE control plane1Node launch template1Node pool1Persistent volumes and stateful sets1RBAC and service accounts1Container registry and its credentials1Telemetry agent1
Contributing factor
Automation amplification4No safeguard on destructive change4Excessive blast radius3Missed in review3Unverified assumption2Monitoring blind spot2No configuration tests2No runbook2No staged rollout2Untested backup1Unreviewed defaults1Harmful manual action1Unrepresentative staging1
Organisation
Aqua Security1DFDS1DigitalOcean1Grafana Labs1InfluxData1Skyscanner1Spotify1
  1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  2. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  3. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
  4. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  5. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  6. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  7. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019

Faulty deployment1 case

Component
CalicoIngress controller
Contributing factor
Excessive blast radiusMonitoring blind spotNo runbook
Organisation
CircleCI
  1. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021