Cases9 cases

4 to 8 of 9

1. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes

   2i2c2025

   Symptoms

   On 12 May 2025 at 20:15 EEST a Freshdesk ticket reported that users' servers on the Earthscope hub were taking more than 15 minutes to start and that their kernels were being killed. It happened during a workshop and also affected other instructors using the hub; a second report said a short course of about 30 users was having major problems.

   Pod eviction

   Trigger

   A workshop with about 30 users on a profile option that guaranteed 8 GB of memory and set no memory limit on 32 GB nodes.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitNode pool

   Contributing factor

   No resource requests or limits

   Detection

   A Freshdesk support ticket from the community at 20:15 EEST, brought to the team's attention in the #support-freshdesk Slack channel at 20:49.

   Remediation

   • ran a helm rollout and hub redeploy, which partly improved server starts
   • made the 'medium' profile available and raised its memory guarantee to reduce overcommitment
   • pre-warmed the cluster with 15 nodes for the 30-user workshop
   • set explicit memory limits, with the guarantee equal to the limit, on the community's small-node profile options (https://github.com/2i2c-org/infrastructure/pull/6047)

   Action items

   • Write an incident report
   • Hold an internal retrospective about the incident (https://github.com/2i2c-org/meta/issues/2194)

   Lessons

   A memory guarantee without a limit lets a user server grow into memory that other pods on the same node were counting on, and on a shared node that ends with users' pods evicting each other. Setting the guarantee equal to the limit removes the overcommitment. For a known workshop, pre-warming enough nodes ahead of time keeps server start times short.

   View case
2. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up

   2i2c2023

   Symptoms

   The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

   Pod evictionPods Pending

   Trigger

   A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode pool

   Contributing factor

   No headroomExcessive blast radiusMonitoring blind spot

   Detection

   A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

   Action items

   • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
   • Make the Grafana OOMKiller dashboard work (#2213)
   • Move LEAP cloud infrastructure to shared nodes (#2209)
   • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
   • Decide whether Prometheus should scrape dask worker pods (#2366)

   Lessons

   Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

   View case
3. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)

   2i2c2022

   Symptoms

   A user of the LIS JupyterHub reported a "backoff after failed scale-up" error, and user servers would not start. An engineer opened incident #36, "LIS hub cannot scale", at 5:40 AM Pacific on 14 November 2022.

   Scale-up failurePods PendingService outage

   Trigger

   At 10 AM UK time on 14 November 2022, the B zone of Google Cloud's europe-west2 region (London) ran out of resources to allocate. The LIS cluster's nodes were restricted to that zone.

   Root cause

   Cloud provider outage

   Component

   Node poolCluster autoscaler

   Contributing factor

   Single point of failurePages not delivered

   Detection

   A user reported the "backoff after failed scale-up" error; an engineer opened the incident at 5:40 AM Pacific, about three hours and forty minutes after the zone ran out of resources.

   Remediation

   • edited the node pool in the GCP console to allow node creation in the A and C zones of europe-west2, which were not exhausted

   Action items

   • Either restrict the node pool back to the B zone after the Black Friday event, or move the cluster and the NFS Filestore to the A zone (infrastructure issue #1944)
   • Investigate an enterprise plan that allows a regional NFS, at higher cost (infrastructure issue #1945)
   • Make sure PagerDuty notifications always get through to engineers (team-compass issue #574)

   Lessons

   A node pool pinned to one zone for storage performance cannot scale when the cloud provider runs out of capacity in that zone. The autoscaler's "backoff after failed scale-up" message usually points to quotas, and raising quotas does nothing here; the ZONE_RESOURCE_POOL_EXHAUSTED error in the logs is what identifies a zone without capacity. The review found that one engineer wearing many hats (incident commander, comms and debugging), with pages not reaching the other responders, made the response harder; a team response would have given more support.

   View case
4. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up

   2i2c2022

   Symptoms

   On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

   High error ratePersistent volume full

   Trigger

   The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

   Root cause

   Capacity exhaustion

   Component

   DatabasePersistent volumes and stateful setsJupyterHub

   Contributing factor

   No alert on limit

   Detection

   A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

   Remediation

   • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
   • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
   • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
   • Restarted the hub pod, which came back with the larger disk and restored service

   Action items

   • Stop keeping hub logs on the same disk as the hub database
   • Add a Grafana graph of free space on the JupyterHub database disk
   • Alert on elevated error rates

   Lessons

   Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

   View case
5. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable

   2i2c2022

   Symptoms

   On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

   Service outageNode NotReady

   Trigger

   Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system image

   Contributing factor

   Limited debug access

   Detection

   The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

   Remediation

   • Started an upgrade of the cluster's Kubernetes and node versions, which replaced all old nodes with new working ones

   Action items

   • Document how to SSH into the Kubernetes nodes
   • Find a way to be notified of Azure advisories

   Lessons

   Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

   View case

By root cause4