Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Excessive blast radius

과도한 영향 범위 · blast-radius-too-wide

32 cases · 2017 to 2026

Root cause Configuration change11 Capacity exhaustion7 Software defect5 Faulty deployment3 Cloud provider outage2 Dependency failure2 Credential exposure1 Breaking change in an upgrade1

Component Application database5 Ingress controller5 Kubernetes API server5 Cloud load balancer4

Symptom Service outage14 High error rate10 Node NotReady6 Control plane unresponsive5

Organisation OpenAI3 2i2c2 DFDS2 Microsoft Azure2 Zalando2

Cases

IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026
Root cause Configuration change
IL-0069 Attackers stole a high-privilege service account token from a production Kubernetes cluster and used it to reach a cryptocurrency exchange's financial systemsUnnamed cryptocurrency exchange · 2026
Root cause Credential exposure

View on the graph

Container memory limit4

Telemetry agent4

Admission webhook3

Configuration templating2

Container runtime2

Client-side DNS resolver and cache2

Container registry and its credentials2

Service discovery2

AKS control plane1

Cluster dashboard1

conntrack table1

Credential broker for pods1

GKE control plane1

Liveness and readiness probes1

Jobs and CronJobs1

Kubernetes controller manager1

Application logging library1

Model request routing1

Node operating system image1

Pod scheduling and placement1

RBAC and service accounts1

Resources deleted5

Monitoring outage4

Unauthorised access or a foreign workload4

Container crash loop3

Probe failures3

DNS resolution failures2

Image pull failures2

Intermittent timeouts2

Sustained high node CPU2

Node recreation loop2

Connection pool exhaustion1

Connection resets1

Credentials unavailable1

Database overload1

Intermittent connection failures1

Load imbalance1

UK Ministry of Justice1

Unnamed cryptocurrency exchange1

IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop

Yonghyun Kim · 2025

Root cause Dependency failure

IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error ratesOpenAI · 2025

Root cause Configuration change

IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managedOpenAI · 2025

Root cause Faulty deployment

IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)Wiz · 2025

Root cause Software defect

IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025

Root cause Configuration change

IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clustersOpenAI · 2024

Root cause Configuration change

IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

Root cause Capacity exhaustion

IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024

Root cause Configuration change

IL-0065 Any pod could take over the cluster by rewriting Argo CD's unauthenticated Redis cache (CVE-2024-31989)Cycode · 2024

Root cause Software defect

IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia SoutheastMicrosoft Azure · 2023

Root cause Cloud provider outage

IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023

Root cause Configuration change

IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

Root cause Capacity exhaustion

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022

Root cause Configuration change

IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022

Root cause Breaking change in an upgrade

IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutesGoogle Cloud · 2022

Root cause Software defect

IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021

Root cause Software defect

IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021

Root cause Configuration change

IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.comCircleCI · 2021

Root cause Faulty deployment

IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configuredDevsisters · 2021

Root cause Cloud provider outage

IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020

Root cause Capacity exhaustion

IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of trafficAirbnb · 2019

Root cause Faulty deployment

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

Root cause Configuration change

IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019

Root cause Capacity exhaustion

IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019

Root cause Configuration change

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change

IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1

Root cause Dependency failure

IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

Root cause Capacity exhaustion

IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018

Root cause Capacity exhaustion

IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod evictionOath · 2018

Root cause Software defect

IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017

Root cause Capacity exhaustion