Cases3 cases

1 to 3 of 3

1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created

   2i2c2026

   Symptoms

   At 15:10 (Bucharest time) on 2026-03-31 an alert fired for the prod hub on the berkeley-geojupyter cluster on AWS: "[FIRING:1] At least two servers failed to start in the last 30m cluster=berkeley-geojupyter hub=prod (provider=aws immediate action needed)". It opened PagerDuty incident #2057.

   Scale-up failure

   Trigger

   An upgrade merged on 2026-03-30 bumped the cluster autoscaler version, and the new version was incompatible with Kubernetes 1.33, which the berkeley-geojupyter cluster still ran.

   Root cause

   Breaking change in an upgrade

   Component

   Cluster autoscaler

   Contributing factor

   Deferred upgradesUnverified assumption

   Detection

   An alert that at least two servers failed to start in the last 30 minutes fired at 15:10 and opened PagerDuty incident #2057. The report notes it was triggered by an automatic health check run, not by users being unable to start servers.

   Remediation

   • upgraded the cluster's Kubernetes version from 1.33 to 1.34
   • downgraded the cluster autoscaler by one patch version to 9.55.0, the version meant for Kubernetes 1.34

   Action items

   • The change in https://github.com/2i2c-org/infrastructure/pull/8008 (the report gives only the link)
   • Use recurrent issues, such as the recurrent-k8s-aws-upgrades workflow, to standardize Kubernetes versions across clusters

   Lessons

   Because the context of the autoscaler bump was fresh, the team went down the right path from the start. The upgrade batches had been designed with only 1.32 clusters in mind, so the one 1.33 cluster was missed and got an incompatible autoscaler. 2i2c notes it was lucky that an automatic health check, not real users, hit the failure first.

   View case
2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates

   2i2c2026

   Symptoms

   An attempt to launch a new hub failed because its ingress could not obtain a new TLS certificate. The day before, the same engineer had recreated the support deployment on the 2i2c-aws-us cluster and seen its TLS certificate orders stuck pending.

   Misrouted requestsDeploys failing

   Trigger

   Creating new hub deployments that needed new TLS certificates on clusters recently migrated from ingress-nginx to nginx-ingress.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllercert-manager

   Contributing factor

   Change with a delayed effect

   Detection

   An attempt to launch a new hub failed; the engineer had seen certificate orders stuck pending on another cluster the day before and declared an incident.

   Remediation

   • annotated the Ingress objects that need TLS so that cert-manager could edit them in place to complete the ACME challenge

   Action items

   • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/8016
   • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/3278

   Lessons

   Swapping one ingress controller for another can change how Ingress objects that share a host are combined. cert-manager's temporary challenge Ingress depends on that behaviour, so the breakage shows up when a deployment needs a new certificate. 2i2c notes that it got lucky: it had already learned of this possible cause while testing on an internal cluster, and changing the cert-manager configuration was quick.

   View case
3. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable

   2i2c2022

   Symptoms

   On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

   Service outageNode NotReady

   Trigger

   Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system image

   Contributing factor

   Limited debug access

   Detection

   The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

   Remediation

   • Started an upgrade of the cluster's Kubernetes and node versions, which replaced all old nodes with new working ones

   Action items

   • Document how to SSH into the Kubernetes nodes
   • Find a way to be notified of Azure advisories

   Lessons

   Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

   View case

By root cause1