Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

No safeguard on destructive change

파괴적 변경 보호 장치 부재 · no-safeguard-on-destructive-change

12 cases · 2019 to 2026

Root cause Configuration change4 Operator error4 Software defect3 Faulty deployment1

Component Ingress controller3 Argo CD2 Cloud load balancer2 Configuration templating2

Symptom Resources deleted8 High error rate6 Service outage6 Intermittent timeouts2

Organisation OpenAI3 Grafana Labs2 2i2c1 Argo CD users1 DigitalOcean

Cases

IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSyncArgo CD users · 2026
Root cause Software defect
IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error ratesOpenAI · 2026
Root cause Software defect
IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour

View on the graph

Persistent volumes and stateful sets2

Automated capacity manager1

Credential broker for pods1

Application database1

GKE control plane1

Jobs and CronJobs1

Kubernetes API server1

Control plane unresponsive1

Monitoring outage1

Stale endpoint routing1

1

UK Ministry of Justice1

2i2c · 2025

Root cause Operator error

IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managedOpenAI · 2025

Root cause Faulty deployment

IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025

Root cause Configuration change

IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024

Root cause Configuration change

IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022

Root cause Operator error

IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022

Root cause Configuration change

IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022

Root cause Operator error

IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dnsUK Ministry of Justice · 2020

Root cause Operator error

IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020

Root cause Software defect

IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019

Root cause Configuration change