Cases9 cases

8 to 9 of 9

1. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable

   2i2c2022

   Symptoms

   On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

   Service outageNode NotReady

   Trigger

   Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system image

   Contributing factor

   Limited debug access

   Detection

   The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

   Remediation

   • Started an upgrade of the cluster's Kubernetes and node versions, which replaced all old nodes with new working ones

   Action items

   • Document how to SSH into the Kubernetes nodes
   • Find a way to be notified of Azure advisories

   Lessons

   Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

   View case
2. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub

   2i2c2020

   Symptoms

   At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.

   Service outageSustained high node CPU

   Trigger

   WER students' pods used a good chunk of their memory limit (tests had shown they almost always use just under 1G of their 2G limit, against a 512M guarantee) on nodes shared with the hub's core pods.

   Root cause

   Capacity exhaustion

   Component

   Node poolContainer memory limit

   Contributing factor

   No resource requests or limitsUnverified assumptionExcessive blast radius

   Detection

   WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).

   Remediation

   • removed memory overcommit for the WER students
   • raised resource guarantees for all core pods, which restarted them and moved some to a new node

   Action items

   • Put user pods in a separate pool so they do not create pressure on the core pods (infrastructure#89)
   • Set limits on the support infrastructure (Prometheus, Grafana, ingress) as well (infrastructure#90)
   • Document and think about overcommit ratios for memory usage (infrastructure#91)
   • Set up better Grafana dashboards to monitor resource usage (infrastructure#92)
   • Document how people can get kubectl access to the cluster so others can look into issues too (infrastructure#87)

   Lessons

   Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.

   View case

By root cause4