Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases19 cases

16 to 19 of 19

  1. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022
    Symptoms

    On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

    Service outageNode NotReady
    Trigger

    Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system image
    Contributing factor
    Limited debug access
    Detection

    The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

    Remediation
    • Started an upgrade of the cluster's Kubernetes and node versions, which replaced all old nodes with new working ones
    Action items
    • Document how to SSH into the Kubernetes nodes
    • Find a way to be notified of Azure advisories
    Lessons

    Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

    View case
  2. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021
    Symptoms

    At 11:32 UTC on 18 November 2021, a user reported in the jenkinsci/jenkins Gitter channel that https://www.jenkins.io/doc/pipeline/steps/ was answering HTTP/503 errors with the message "Guru meditation" from Fastly's varnish.

    Service outageResources deleted
    Trigger

    The ingress-nginx Helm chart was upgraded from 4.0.6 to 4.0.8 at 10:42 UTC, and the new chart added an admission webhook rule that forbids Nginx configuration blocks in Ingress annotations.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerAdmission webhookHelmCDN and edge network
    Contributing factor
    Unverified assumptionCache-masked failureMonitoring blind spotHarmful manual action
    Detection

    A user reported in the jenkinsci/jenkins Gitter channel at 11:32 UTC that a jenkins.io page answered HTTP/503 errors, and status.jenkins.io reported the incident at 11:49 UTC.

    Remediation
    • Disabled the automated systems, rolled the ingress-nginx chart back to 4.0.6 and deleted the admission webhook by hand
    • Removed the forbidden annotations from the jenkins.io configuration
    • Upgraded the jenkins.io Helm chart to the last version known to work with ingress-nginx 4.0.8
    • Redeployed the jenkins.io Helm chart through the automated system and told Fastly to drop all cached pages
    Action items
    • Always read the changelog in depth when updating the ingress controllers, with two people pairing in real time
    • Add custom Datadog monitoring for www.origin.jenkins.io so that such an outage is detected
    • Make the monitor send the Host header www.jenkins.io so it is not redirected to Fastly
    • Check a set of pages, such as the pipeline steps page, not only the front page
    • Consider an ephemeral k3s end-to-end test before deploying to production
    Lessons

    Trusting a patch version of a Helm chart for the Ingress controller without reading its changelog in depth let a new admission rule remove the jenkins.io Ingress in production. Without monitoring on origin.jenkins.io, a user found the outage. A complete staging cluster with the same DNS setup would have caught the problem before production.

    View case
  3. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
    Symptoms

    At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

    Deploys failing
    Trigger

    The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

    Root cause
    Breaking change in an upgrade
    Component
    Admission webhookIngress controller
    Contributing factor
    Unrepresentative staging
    Detection

    A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

    Remediation
    • rolled back the ModSec ingress controller module to v0.0.7
    Action items
    • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
    • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
    • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
    • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)
    View case
  4. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable
    Enigmo2020
    Symptoms

    Two or three days after Enigmo created EKS managed node groups and moved its pods there from the old node groups, the cluster suddenly became unstable: pods restarted again and again or could not stop and stayed around indefinitely, node CPU and memory load rose high, and the nodes became unusable from the cluster.

    Pods stuck in TerminatingSustained high node CPUMemory pressureNode NotReadyUpgrade stall
    Trigger

    Enigmo created EKS managed node groups that used an AMI containing the bug and moved its pods onto them, as preparation for upgrading EKS from 1.15 to 1.16.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imageNode pool
    Contributing factor
    Ineffective rollbackChange with a delayed effect
    Remediation
    • Changed the managed node groups' AMI version to the fixed release
    • Ran the managed node group update through CloudFormation
    • Watched the pods across all namespaces during the update
    • Force-deleted every pod that stayed in Terminating with kubectl delete --grace-period=0 --force
    Action items
    • Check the latest releases of the cluster's modules quarterly or monthly and upgrade them regularly
    Lessons

    As long as a team uses Amazon EKS or Kubernetes, it has to keep track of upgrades of the cluster and of modules such as ingress controllers, because a release with breaking changes can be announced without warning after a long stable period. Checking releases and upgrading on a regular schedule keeps the effect of each upgrade on the service small.

    View case
Check your cluster for this

These 19 cases involve Ingress controller and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade19 cases

Component
Ingress controller6Node operating system image6Admission webhook5Cloud load balancer3Client-side DNS resolver and cache3cert-manager2ConfigMaps and Secrets2GKE control plane2Helm2Istio2kube-proxy2kubelet2Linux kernel2Node pool2Calico1CDN and edge network1Cilium1Cluster autoscaler1CoreDNS1etcd1Model inference server1Karpenter1Kubernetes API server1Persistent volumes and stateful sets1Container registry and its credentials1systemd-networkd1Web application firewall rules1
Contributing factor
Change with a delayed effect5Monitoring blind spot5Unrepresentative staging4No staged rollout3Ineffective rollback3Deferred upgrades3Unverified assumption2Automation amplification2Harmful manual action2Single point of failure2Untested backup1Excessive blast radius1Configuration not in code1Unreviewed defaults1Ignored early signal1Cache-masked failure1Unpatched known bug1Limited debug access1No headroom1Partial rollout1Unpinned version1
Organisation
2i2c3PortSwigger2UK Ministry of Justice2Altinity1CircleCI1Datadog1Enigmo1GitHub1Honeycomb1Jenkins project1Microsoft Azure1Reddit1Sunwoo Park1Wealthy1Wikimedia Foundation1
  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  3. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  4. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026
  5. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  6. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  7. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  8. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
  9. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024
  10. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  11. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  12. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  13. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  14. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  15. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
  16. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022
  17. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021
  18. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
  19. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable
    Enigmo2020