Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Root cause

Operator error

운영 실수 · human-operation

7 cases · 2019 to 2025

Component Ingress controller4 Node pool2 Persistent volumes and stateful sets2 CI system1 Cloud load balancer1 Credential broker for pods1 Application database1 Elasticsearch or OpenSearch1 external-dns1 Helm1 Jobs and CronJobs1 kube-proxy1 Node launch template1 Model hub1 Prometheus1 Container registry and its credentials1 Terraform1

Contributing factor No safeguard on destructive change4 Unverified assumption2 Credential in configuration1 Unreviewed defaults1

Symptom Service outage5 Resources deleted3 High error rate2 Image pull failures1

Organisation UK Ministry of Justice2 2i2c1 DevOps Hof1 GitLab1 OpenAI

Cases

IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025
IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull imagesUK Ministry of Justice · 2023
IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022
IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022
IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0Palark · 2021

View on the graph

Overlapping failures1

Shared credential1

Deferred upgrades1

Intermittent timeouts1

Monitoring outage1

Probe failures1

Stale endpoint routing1

1

IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns

UK Ministry of Justice · 2020

IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx podsDevOps Hof · 2019