Incident Library
Applied filters Root causeBreaking change in an upgradeComponentAdmission webhookDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases5 cases

1 to 5 of 5

  1. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
    Symptoms

    From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

    Service outageTLS certificate errors
    Trigger

    The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer
    Contributing factor
    Change with a delayed effectMonitoring blind spot
    Detection

    Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

    Remediation
    • Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
    • Redeployed the AWS Load Balancer Controller configuration with Helm
    • Verified that new certificates were generated and the webhooks worked again
    Action items
    • Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
    • Add high-level application access alerting for each SaaS cluster from an external probe
    View case
  2. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
    Symptoms

    On 2023-05-18 a Slack alert arrived at 20:01 and Istio stopped working properly in the production GKE cluster. The istiod error logs said that the v1beta1.Ingress and v1beta1.ValidatingWebhookConfiguration resources could not be found.

    Service outage
    Trigger

    GKE automatically upgraded the production cluster's control plane from 1.22.17-gke.7500 to 1.23.17-gke.1700 at 19:52 on 2023-05-18.

    Root cause
    Breaking change in an upgrade
    Component
    IstioGKE control planeAdmission webhook
    Contributing factor
    Deferred upgrades
    Detection

    A Slack alert at 20:01 on 2023-05-18, nine minutes after the control plane upgrade. The istiod error logs showed that the v1beta1 Ingress and ValidatingWebhookConfiguration resources could not be found.

    Remediation
    • upgraded Istio from 1.7.4 to 1.13.3
    Lessons

    The first diagnosis, end of support for Istio, was wrong; the real cause was found by matching the alert time against the cluster's UPGRADE_MASTER operations in Logs Explorer. The author recommends checking the version compatibility matrices of Kubernetes, Istio and other packages, keeping a production version for at least six months and upgrading three months before end of support, keeping the test environment on the latest version, backing up before upgrades, managing deployments and versions with a tool such as Argo CD, and testing upgrades in a test environment before production.

    View case
  3. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
    Symptoms

    About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

    Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag
    Trigger

    An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

    Root cause
    Breaking change in an upgrade
    Component
    CalicoKubernetes API serverAdmission webhooketcd
    Contributing factor
    Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure
    Detection

    The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

    Remediation
    • deleted the Open Policy Agent webhook configurations to stop API server timeouts
    • terminated all worker nodes of the cluster
    • restored the control plane from the pre-upgrade backup onto the first control plane node
    • switched off the cluster autoscaler until the control plane was stable
    • generated new TLS certificates so replacement control plane nodes could join etcd
    • brought traffic back in steps from 1% to 100%
    • restarted legacy services that had been stopped by hand
    Action items
    • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
    • Standardise components across clusters and replace bespoke configurations
    • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
    • Add pre-production clusters that can be tested with confidence
    • Improve the backup restore process and make cluster upgrades safe
    Lessons

    Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

    View case
  4. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021
    Symptoms

    At 11:32 UTC on 18 November 2021, a user reported in the jenkinsci/jenkins Gitter channel that https://www.jenkins.io/doc/pipeline/steps/ was answering HTTP/503 errors with the message "Guru meditation" from Fastly's varnish.

    Service outageResources deleted
    Trigger

    The ingress-nginx Helm chart was upgraded from 4.0.6 to 4.0.8 at 10:42 UTC, and the new chart added an admission webhook rule that forbids Nginx configuration blocks in Ingress annotations.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerAdmission webhookHelmCDN and edge network
    Contributing factor
    Unverified assumptionCache-masked failureMonitoring blind spotHarmful manual action
    Detection

    A user reported in the jenkinsci/jenkins Gitter channel at 11:32 UTC that a jenkins.io page answered HTTP/503 errors, and status.jenkins.io reported the incident at 11:49 UTC.

    Remediation
    • Disabled the automated systems, rolled the ingress-nginx chart back to 4.0.6 and deleted the admission webhook by hand
    • Removed the forbidden annotations from the jenkins.io configuration
    • Upgraded the jenkins.io Helm chart to the last version known to work with ingress-nginx 4.0.8
    • Redeployed the jenkins.io Helm chart through the automated system and told Fastly to drop all cached pages
    Action items
    • Always read the changelog in depth when updating the ingress controllers, with two people pairing in real time
    • Add custom Datadog monitoring for www.origin.jenkins.io so that such an outage is detected
    • Make the monitor send the Host header www.jenkins.io so it is not redirected to Fastly
    • Check a set of pages, such as the pipeline steps page, not only the front page
    • Consider an ephemeral k3s end-to-end test before deploying to production
    Lessons

    Trusting a patch version of a Helm chart for the Ingress controller without reading its changelog in depth let a new admission rule remove the jenkins.io Ingress in production. Without monitoring on origin.jenkins.io, a user found the outage. A complete staging cluster with the same DNS setup would have caught the problem before production.

    View case
  5. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
    Symptoms

    At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

    Deploys failing
    Trigger

    The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

    Root cause
    Breaking change in an upgrade
    Component
    Admission webhookIngress controller
    Contributing factor
    Unrepresentative staging
    Detection

    A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

    Remediation
    • rolled back the ModSec ingress controller module to v0.0.7
    Action items
    • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
    • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
    • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
    • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)
    View case
Check your cluster for this

These 5 cases involve Admission webhook and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade5 cases

Component
Admission webhook5Ingress controller3Helm2Calico1CDN and edge network1cert-manager1Cloud load balancer1ConfigMaps and Secrets1etcd1GKE control plane1Istio1Kubernetes API server1
Contributing factor
Monitoring blind spot3Unrepresentative staging2Unverified assumption1Untested backup1Configuration not in code1Change with a delayed effect1Cache-masked failure1Harmful manual action1Single point of failure1Deferred upgrades1
Organisation
Jenkins project1PortSwigger1Reddit1Sunwoo Park1UK Ministry of Justice1
  1. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  2. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  3. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  4. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021
  5. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021