Incident Library
Applied filters Root causeBreaking change in an upgradeDomainKubernetesOrganisation2i2cClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
    Symptoms

    At 15:10 (Bucharest time) on 2026-03-31 an alert fired for the prod hub on the berkeley-geojupyter cluster on AWS: "[FIRING:1] At least two servers failed to start in the last 30m cluster=berkeley-geojupyter hub=prod (provider=aws immediate action needed)". It opened PagerDuty incident #2057.

    Scale-up failure
    Trigger

    An upgrade merged on 2026-03-30 bumped the cluster autoscaler version, and the new version was incompatible with Kubernetes 1.33, which the berkeley-geojupyter cluster still ran.

    Root cause
    Breaking change in an upgrade
    Component
    Cluster autoscaler
    Contributing factor
    Deferred upgradesUnverified assumption
    Detection

    An alert that at least two servers failed to start in the last 30 minutes fired at 15:10 and opened PagerDuty incident #2057. The report notes it was triggered by an automatic health check run, not by users being unable to start servers.

    Remediation
    • upgraded the cluster's Kubernetes version from 1.33 to 1.34
    • downgraded the cluster autoscaler by one patch version to 9.55.0, the version meant for Kubernetes 1.34
    Action items
    • The change in https://github.com/2i2c-org/infrastructure/pull/8008 (the report gives only the link)
    • Use recurrent issues, such as the recurrent-k8s-aws-upgrades workflow, to standardize Kubernetes versions across clusters
    Lessons

    Because the context of the autoscaler bump was fresh, the team went down the right path from the start. The upgrade batches had been designed with only 1.32 clusters in mind, so the one 1.33 cluster was missed and got an incompatible autoscaler. 2i2c notes it was lucky that an automatic health check, not real users, hit the failure first.

    View case
  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
    Symptoms

    An attempt to launch a new hub failed because its ingress could not obtain a new TLS certificate. The day before, the same engineer had recreated the support deployment on the 2i2c-aws-us cluster and seen its TLS certificate orders stuck pending.

    Misrouted requestsDeploys failing
    Trigger

    Creating new hub deployments that needed new TLS certificates on clusters recently migrated from ingress-nginx to nginx-ingress.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllercert-manager
    Contributing factor
    Change with a delayed effect
    Detection

    An attempt to launch a new hub failed; the engineer had seen certificate orders stuck pending on another cluster the day before and declared an incident.

    Remediation
    • annotated the Ingress objects that need TLS so that cert-manager could edit them in place to complete the ACME challenge
    Action items
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/8016
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/3278
    Lessons

    Swapping one ingress controller for another can change how Ingress objects that share a host are combined. cert-manager's temporary challenge Ingress depends on that behaviour, so the breakage shows up when a deployment needs a new certificate. 2i2c notes that it got lucky: it had already learned of this possible cause while testing on an internal cluster, and changing the cert-manager configuration was quick.

    View case
  3. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022
    Symptoms

    On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

    Service outageNode NotReady
    Trigger

    Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system image
    Contributing factor
    Limited debug access
    Detection

    The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

    Remediation
    • Started an upgrade of the cluster's Kubernetes and node versions, which replaced all old nodes with new working ones
    Action items
    • Document how to SSH into the Kubernetes nodes
    • Find a way to be notified of Azure advisories
    Lessons

    Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

    View case
Check your cluster for this

These 3 cases involve cert-manager and Cluster autoscaler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade3 cases

Component
cert-manager1Cluster autoscaler1Ingress controller1Node operating system image1
Contributing factor
Unverified assumption1Change with a delayed effect1Limited debug access1Deferred upgrades1
Organisation
2i2c3
  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  3. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022