Incident Library
Applied filters ComponentAdmission webhookDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases13 cases

13 to 13 of 13

  1. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
    Symptoms

    Users reported that their jobs were not starting, with some sign that image pulls were involved. Many pods across the cluster were in error states related to image pulls, while applications that were already running were not affected.

    Image pull failuresService outage
    Trigger

    A change to the permissions on a storage bucket that one application needed to read. The application could no longer start, and its DaemonSet, running on roughly a thousand nodes of the cluster, went into a crash loop with imagePullPolicy: Always, so every restart pulled the image again although it was already on the node.

    Root cause
    Capacity exhaustion
    Component
    Container registry and its credentialsNAT gatewayDatabaseAdmission webhook
    Contributing factor
    Automation amplificationSingle point of failureUnknown limitHarmful manual actionExcessive blast radius
    Detection

    Users reported that their jobs were not starting; monitoring had not caught it first.

    Remediation
    • replaced the affected NAT instances so registry traffic came from new source IPs
    • reconfigured the static IPs and firewall rules so applications could reach the Cloud SQL database again
    Action items
    • Add an admission webhook that rejects floating image tags, starting with latest
    • Apply the image tag webhook to workload objects only, not to pods, after it blocked pods created by in-cluster controllers
    Lessons

    The speakers' takeaway is to be very careful with DaemonSets in large clusters: they say they had broken everything DaemonSets depend on, including DNS, Vault and the image repository, and that it is very easy to break things when managing them by the thousand. imagePullPolicy: Always made every crash-loop restart pull the image again. Sending all registry traffic through three NAT instances put the whole region behind three source IPs, which the registry rate-limited, and changing those IPs cut the connections to a critical database reached over the internet through them. An admission webhook on pods rejects the pods that controllers in the cluster create, where no user sees the error.

    View case
Check your cluster for this

These 13 cases involve Admission webhook and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause7

Breaking change in an upgrade5 cases

Component
Admission webhook5Ingress controller3Helm2Calico1CDN and edge network1cert-manager1Cloud load balancer1ConfigMaps and Secrets1etcd1GKE control plane1Istio1Kubernetes API server1
Contributing factor
Monitoring blind spot3Unrepresentative staging2Unverified assumption1Untested backup1Configuration not in code1Change with a delayed effect1Cache-masked failure1Harmful manual action1Single point of failure1Deferred upgrades1
Organisation
Jenkins project1PortSwigger1Reddit1Sunwoo Park1UK Ministry of Justice1
  1. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  2. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  3. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  4. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021
  5. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021

Software defect2 cases

Component
Admission webhook2Horizontal pod autoscaler1Ingress controller1
Contributing factor
Excessive blast radius1Unreviewed defaults1Change with a delayed effect1Ignored early signal1Monitoring blind spot1
Organisation
Airbnb1Wiz1
  1. IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)
    Wiz2025
  2. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020

Dependency failure1 case

Component
Admission webhookIstioNode operating system image
Contributing factor
Unreviewed defaultsUnknown limit
Organisation
Tetrate (unnamed U.S. federal customer)
  1. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod
    Tetrate (unnamed U.S. federal customer)2025

Certificate or credential expiry1 case

Component
Admission webhookPolicy engine
Contributing factor
Unreviewed defaultsIgnored early signalMonitoring blind spotDeferred upgrades
Organisation
DFDS
  1. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022

Configuration change2 cases

Component
Admission webhook2Kubernetes API server2GKE control plane1kubelet1
Contributing factor
Monitoring blind spot2Unverified assumption1Automation amplification1Excessive blast radius1No configuration tests1No health probes1
Organisation
DFDS1Jetstack1
  1. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  2. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019

Faulty deployment1 case

Component
Admission webhookIngress controller
Contributing factor
Single point of failure
Organisation
UK Ministry of Justice
  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020

Capacity exhaustion1 case

Component
Admission webhookDatabaseNAT gatewayContainer registry and its credentials
Contributing factor
Automation amplificationExcessive blast radiusUnknown limitHarmful manual actionSingle point of failure
Organisation
Datadog
  1. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019