Incident Library
Applied filters Root causeBreaking change in an upgradeDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases19 cases

1 to 5 of 19

  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
    Symptoms

    At 15:10 (Bucharest time) on 2026-03-31 an alert fired for the prod hub on the berkeley-geojupyter cluster on AWS: "[FIRING:1] At least two servers failed to start in the last 30m cluster=berkeley-geojupyter hub=prod (provider=aws immediate action needed)". It opened PagerDuty incident #2057.

    Scale-up failure
    Trigger

    An upgrade merged on 2026-03-30 bumped the cluster autoscaler version, and the new version was incompatible with Kubernetes 1.33, which the berkeley-geojupyter cluster still ran.

    Root cause
    Breaking change in an upgrade
    Component
    Cluster autoscaler
    Contributing factor
    Deferred upgradesUnverified assumption
    Detection

    An alert that at least two servers failed to start in the last 30 minutes fired at 15:10 and opened PagerDuty incident #2057. The report notes it was triggered by an automatic health check run, not by users being unable to start servers.

    Remediation
    • upgraded the cluster's Kubernetes version from 1.33 to 1.34
    • downgraded the cluster autoscaler by one patch version to 9.55.0, the version meant for Kubernetes 1.34
    Action items
    • The change in https://github.com/2i2c-org/infrastructure/pull/8008 (the report gives only the link)
    • Use recurrent issues, such as the recurrent-k8s-aws-upgrades workflow, to standardize Kubernetes versions across clusters
    Lessons

    Because the context of the autoscaler bump was fresh, the team went down the right path from the start. The upgrade batches had been designed with only 1.32 clusters in mind, so the one 1.33 cluster was missed and got an incompatible autoscaler. 2i2c notes it was lucky that an automatic health check, not real users, hit the failure first.

    View case
  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
    Symptoms

    An attempt to launch a new hub failed because its ingress could not obtain a new TLS certificate. The day before, the same engineer had recreated the support deployment on the 2i2c-aws-us cluster and seen its TLS certificate orders stuck pending.

    Misrouted requestsDeploys failing
    Trigger

    Creating new hub deployments that needed new TLS certificates on clusters recently migrated from ingress-nginx to nginx-ingress.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllercert-manager
    Contributing factor
    Change with a delayed effect
    Detection

    An attempt to launch a new hub failed; the engineer had seen certificate orders stuck pending on another cluster the day before and declared an incident.

    Remediation
    • annotated the Ingress objects that need TLS so that cert-manager could edit them in place to complete the ACME challenge
    Action items
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/8016
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/3278
    Lessons

    Swapping one ingress controller for another can change how Ingress objects that share a host are combined. cert-manager's temporary challenge Ingress depends on that behaviour, so the breakage shows up when a deployment needs a new certificate. 2i2c notes that it got lucky: it had already learned of this possible cause while testing on an internal cluster, and changing the cert-manager configuration was quick.

    View case
  3. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
    Symptoms

    At 08:56 on 30 March 2026, about an hour and three quarters after the rollout of a new ingress-nginx to production began, the MoJ Cloud Platform team noticed messages in the #hmpps-auth-audit-registers Slack channel about ModSecurity unexpectedly blocking requests. Users in HMPPS also raised incidents with the MoJ service desk.

    High error rateService outage
    Trigger

    The Cloud Platform team began deploying ingress-nginx 1.14.3, on images maintained by Chainguard, to the three production ingresses at 07:11 on 30 March 2026.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerWeb application firewall rules
    Contributing factor
    Unrepresentative staging
    Detection

    At 08:56 the Cloud Platform team saw messages in the #hmpps-auth-audit-registers Slack channel showing ModSecurity blocking requests unexpectedly; HMPPS users also raised incidents with the MoJ service desk.

    Remediation
    • rolled the ingress-nginx deployment back to the previous version, from 09:14 to 09:43
    • HMPPS applied ModSecurity rule exclusions for the rules blocking traffic to HMPPS Auth
    • added a temporary global exclusion for the CRS rule triggered by the PostHog cookie
    • redeployed ingress-nginx 1.14.3 with both exclusions in place, completed at 14:00
    Action items
    • Rescope the PostHog cookie so it is no longer sent with requests to all *.justice.gov.uk services
    • Remove the temporary global exclusion once the PostHog cookie has expired from enough users' browsers
    • Remind service teams to run dev and test workloads through the non-production ingress controllers so future ModSecurity and CRS updates are tested
    • Run production and non-production workloads in separate clusters in Cloud Platform 3.0, so non-production workloads must use the non-production ingress
    Lessons

    An ingress controller upgrade that bundles a web application firewall also changes which requests are allowed, not only how they are routed. A beta or non-production ingress only catches false positives for traffic that actually passes through it, and teams that keep their test workloads on the production ingress miss them. A fix on the client side, such as rescoping a cookie, takes effect only when old cookies have expired in users' browsers, so the rule change has to wait for it or ship with an exclusion.

    View case
  4. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026
    Symptoms

    From 17:30 UTC on 23 February 2026, right after the ml-serve-codfw Kubernetes cluster was upgraded, 5XX responses appeared in the metrics of a subset of Wikimedia's machine learning model servers and alerts started to fire. About half of the running model servers were affected and stayed unreachable for roughly 24 hours.

    High error rate
    Trigger

    The two production Machine Learning Kubernetes clusters were upgraded to version 1.31, which included upgrading Istio from 1.15 to 1.24; ml-serve-codfw was done at 17:21 UTC on 23 February and ml-serve-eqiad between 08:00 and 11:00 UTC on 24 February.

    Root cause
    Breaking change in an upgrade
    Component
    IstioModel inference server
    Contributing factor
    Ignored early signal
    Detection

    Users of the affected model servers first reported the issue in the talk-to-machine-learning Slack channel. Alerts for HTTP 5XX had fired in the Machine Learning IRC channel right after the codfw upgrade, but nobody on the team noticed them.

    Remediation
    • Deployed an Istio hot patch that restored the broken functionality and fixed most of the HTTP 500 errors
    • Deployed the permanent fix to all affected clusters after verifying it on ml-staging-codfw
    Action items
    • Deliver alerts to the relevant Slack channels and watch IRC or alerts.wikimedia.org
    • Add more SLOs and fix the existing ones, whose Istio metrics changed
    Lessons

    The problem could have been caught on the staging cluster first, or by ad-hoc testing within the team after each cluster upgrade, which had been requested when the upgrades were announced. Not everyone involved knew that httpbb tests existed, and nobody on the Machine Learning team saw the alerts that fired right after the codfw upgrade. Recovery was fairly quick once users reported the problem, helped by having the right person online to find the Istio hot patch.

    View case
  5. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
    Symptoms

    From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

    Service outageTLS certificate errors
    Trigger

    The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer
    Contributing factor
    Change with a delayed effectMonitoring blind spot
    Detection

    Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

    Remediation
    • Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
    • Redeployed the AWS Load Balancer Controller configuration with Helm
    • Verified that new certificates were generated and the webhooks worked again
    Action items
    • Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
    • Add high-level application access alerting for each SaaS cluster from an external probe
    View case
Check your cluster for this

These 19 cases involve Ingress controller and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade19 cases

Component
Ingress controller6Node operating system image6Admission webhook5Cloud load balancer3Client-side DNS resolver and cache3cert-manager2ConfigMaps and Secrets2GKE control plane2Helm2Istio2kube-proxy2kubelet2Linux kernel2Node pool2Calico1CDN and edge network1Cilium1Cluster autoscaler1CoreDNS1etcd1Model inference server1Karpenter1Kubernetes API server1Persistent volumes and stateful sets1Container registry and its credentials1systemd-networkd1Web application firewall rules1
Contributing factor
Change with a delayed effect5Monitoring blind spot5Unrepresentative staging4No staged rollout3Ineffective rollback3Deferred upgrades3Unverified assumption2Automation amplification2Harmful manual action2Single point of failure2Untested backup1Excessive blast radius1Configuration not in code1Unreviewed defaults1Ignored early signal1Cache-masked failure1Unpatched known bug1Limited debug access1No headroom1Partial rollout1Unpinned version1
Organisation
2i2c3PortSwigger2UK Ministry of Justice2Altinity1CircleCI1Datadog1Enigmo1GitHub1Honeycomb1Jenkins project1Microsoft Azure1Reddit1Sunwoo Park1Wealthy1Wikimedia Foundation1
  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  3. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  4. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026
  5. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  6. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  7. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  8. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
  9. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024
  10. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  11. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  12. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  13. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  14. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  15. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
  16. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022
  17. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021
  18. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
  19. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable
    Enigmo2020