Incident Library
Applied filters Root causeBreaking change in an upgradeDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases19 cases

18 to 19 of 19

  1. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
    Symptoms

    At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

    Deploys failing
    Trigger

    The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

    Root cause
    Breaking change in an upgrade
    Component
    Admission webhookIngress controller
    Contributing factor
    Unrepresentative staging
    Detection

    A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

    Remediation
    • rolled back the ModSec ingress controller module to v0.0.7
    Action items
    • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
    • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
    • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
    • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)
    View case
  2. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable
    Enigmo2020
    Symptoms

    Two or three days after Enigmo created EKS managed node groups and moved its pods there from the old node groups, the cluster suddenly became unstable: pods restarted again and again or could not stop and stayed around indefinitely, node CPU and memory load rose high, and the nodes became unusable from the cluster.

    Pods stuck in TerminatingSustained high node CPUMemory pressureNode NotReadyUpgrade stall
    Trigger

    Enigmo created EKS managed node groups that used an AMI containing the bug and moved its pods onto them, as preparation for upgrading EKS from 1.15 to 1.16.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imageNode pool
    Contributing factor
    Ineffective rollbackChange with a delayed effect
    Remediation
    • Changed the managed node groups' AMI version to the fixed release
    • Ran the managed node group update through CloudFormation
    • Watched the pods across all namespaces during the update
    • Force-deleted every pod that stayed in Terminating with kubectl delete --grace-period=0 --force
    Action items
    • Check the latest releases of the cluster's modules quarterly or monthly and upgrade them regularly
    Lessons

    As long as a team uses Amazon EKS or Kubernetes, it has to keep track of upgrades of the cluster and of modules such as ingress controllers, because a release with breaking changes can be announced without warning after a long stable period. Checking releases and upgrading on a regular schedule keeps the effect of each upgrade on the service small.

    View case
Check your cluster for this

These 19 cases involve Ingress controller and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade19 cases

Component
Ingress controller6Node operating system image6Admission webhook5Cloud load balancer3Client-side DNS resolver and cache3cert-manager2ConfigMaps and Secrets2GKE control plane2Helm2Istio2kube-proxy2kubelet2Linux kernel2Node pool2Calico1CDN and edge network1Cilium1Cluster autoscaler1CoreDNS1etcd1Model inference server1Karpenter1Kubernetes API server1Persistent volumes and stateful sets1Container registry and its credentials1systemd-networkd1Web application firewall rules1
Contributing factor
Change with a delayed effect5Monitoring blind spot5Unrepresentative staging4No staged rollout3Ineffective rollback3Deferred upgrades3Unverified assumption2Automation amplification2Harmful manual action2Single point of failure2Untested backup1Excessive blast radius1Configuration not in code1Unreviewed defaults1Ignored early signal1Cache-masked failure1Unpatched known bug1Limited debug access1No headroom1Partial rollout1Unpinned version1
Organisation
2i2c3PortSwigger2UK Ministry of Justice2Altinity1CircleCI1Datadog1Enigmo1GitHub1Honeycomb1Jenkins project1Microsoft Azure1Reddit1Sunwoo Park1Wealthy1Wikimedia Foundation1
  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  3. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  4. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026
  5. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  6. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  7. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  8. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
  9. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024
  10. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  11. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  12. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  13. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  14. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  15. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
  16. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022
  17. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021
  18. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
  19. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable
    Enigmo2020