Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases11 cases

11 to 11 of 11

  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
    Symptoms

    At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.

    Deploys failing
    Trigger

    Version 0.1.0 of the team's ingress controller module started creating a ValidatingWebhookConfiguration resource, and one new 0.1.0 ingress controller in the live-1 cluster failed to be created because of AWS resource limits.

    Root cause
    Faulty deployment
    Component
    Admission webhookIngress controller
    Contributing factor
    Single point of failure
    Detection

    The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.

    Remediation
    • removed by hand all the additional admission controllers created by module version 0.1.0
    • removed the admission webhook from the ingress controller module and released version 0.1.1
    • upgraded every ingress module on version 0.1.0 to 0.1.1
    Lessons

    Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.

    View case
Check your cluster for this

These 11 cases involve Admission webhook and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause7

Certificate or credential expiry3 cases

Component
Admission webhook1Kubernetes API server1kubelet1Node pool1Policy engine1
Contributing factor
Unreviewed defaults1Ignored early signal1Unknown limit1Monitoring blind spot1No alert on limit1Deferred upgrades1
Organisation
dbhang1DFDS1Linode1
  1. IL-0158 An expired credential caused authentication failures that stopped LKE and LKE-E from deploying clusters and nodes in all regions, which also blocked node recycling and cluster autoscaling
    Linode2026
  2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022
  3. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021

Breaking change in an upgrade2 cases

Component
Ingress controller2Admission webhook1cert-manager1
Contributing factor
Change with a delayed effect1Unrepresentative staging1
Organisation
2i2c1UK Ministry of Justice1
  1. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  2. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021

Capacity exhaustion1 case

Component
CoreDNSetcdKubernetes API server
Contributing factor
Unreviewed defaultsHarmful manual action
Organisation
Render
  1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024

Configuration change2 cases

Component
Admission webhook1Kubernetes API server1
Contributing factor
Unverified assumption1Monitoring blind spot1No configuration tests1
Organisation
DFDS1Google Cloud1
  1. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration change
    Google Cloud2024
  2. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020

Cloud provider outage1 case

Component
Cluster autoscalerNode pool
Contributing factor
Limited debug accessMonitoring blind spotOverlapping failures
Organisation
GitLab
  1. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021

Software defect1 case

Component
Admission webhookHorizontal pod autoscaler
Contributing factor
Change with a delayed effectIgnored early signalMonitoring blind spot
Organisation
Airbnb
  1. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020

Faulty deployment1 case

Component
Admission webhookIngress controller
Contributing factor
Single point of failure
Organisation
UK Ministry of Justice
  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020