Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Admission webhook

어드미션 웹훅 · admission-webhook

11 cases · 2019 to 2025

Root cause Breaking change in an upgrade3 Configuration change2 Software defect2 Capacity exhaustion1 Certificate or credential expiry1 Dependency failure1 Faulty deployment1

Contributing factor Monitoring blind spot5 Excessive blast radius3 Unreviewed defaults3 Single point of failure3

Symptom Deploys failing5 Service outage4 Control plane lag2 Container crash loop2

Organisation DFDS2 UK Ministry of Justice2 Airbnb1 Datadog1 Jetstack

Cases, by severity

IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiodTetrate (unnamed U.S. federal customer) · 2025 · sev1
Root cause Dependency failure
IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)Wiz · 2025
Root cause Software defect
A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production

View on the graph

Automation amplification2

Ignored early signal2

Deferred upgrades2

Unrepresentative staging2

Unverified assumption1

Untested backup1

Configuration not in code1

Change with a delayed effect1

Harmful manual action1

No configuration tests1

No health probes1

High error rate2

Control plane unresponsive1

DNS resolution failures1

Image pull failures1

Monitoring outage1

Node recreation loop1

Pods stuck in ContainerCreating1

Probe failures1

Throughput degradation1

Unauthorised access or a foreign workload1

1

Tetrate (unnamed U.S. federal customer)1

IL-0180

Sunwoo Park · 2023

Root cause Breaking change in an upgrade

IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023

Root cause Breaking change in an upgrade

IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutesDFDS · 2022

Root cause Certificate or credential expiry

IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and changeUK Ministry of Justice · 2021

Root cause Breaking change in an upgrade

IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two podsAirbnb · 2020

Root cause Software defect

IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020

Root cause Configuration change

IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 errorUK Ministry of Justice · 2020

Root cause Faulty deployment

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

Root cause Configuration change

IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019

Root cause Capacity exhaustion