Incident Library

IL-0501 2i2c · 2022-08-30kubernetesazurenetworking

Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable

Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes. Hundreds of such updates normally apply automatically without 2i2c noticing, but this one triggered a systemd bug, Ubuntu bug 1988119, which the Microsoft advisory described as a DHCP bug causing DNS errors after the upgrade to systemd 237-3ubuntu10.54. Networking failed on every existing node of the UToronto Kubernetes cluster, all nodes went NotReady, and the hub was down from 11:15 AM to 12:13 PM Pacific time on 30 August 2022, 58 minutes in all.

According to the Ubuntu bug report, a reboot would have fixed the nodes. The nodes were not listed on the Azure Portal VMs page, and an upgrade of the cluster's Kubernetes version was already due and would replace every node, so at 11:53 AM the team started an upgrade of the cluster's Kubernetes and node versions as the fastest way to get fresh nodes. The upgrade retired all old nodes and brought up new working ones, and at 12:14 PM the hub was back online.

Severity
–
Scope
UToronto JupyterHub (jupyter.utoronto.ca) was unreachable; all nodes of its Kubernetes cluster were NotReady
Users affected
all UToronto JupyterHub users
Region
–
Time to detect
–
Time to resolve
58m

Symptoms

On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

At 11:41 AM `kubectl get node` showed every node of the cluster in NotReady, for example "aks-core-34239724-vmss000000 NotReady agent 273d v1.20.7" and "aks-nbdefault-34239724-vmss000010 NotReady agent 193d v1.20.7". Networking had failed on all existing Kubernetes nodes.

At 11:25 AM the customer forwarded a Microsoft advisory: from about 06:00 UTC that day, Azure customers running Ubuntu 18.04 VMs recently upgraded to systemd version 237-3ubuntu10.54 had reported DNS errors when reaching their resources, a bug tracked as Ubuntu bug 1988119, with rebooting the VM to get a fresh DHCP lease and new DNS resolvers as a possible workaround. The `kubernetes.azure.com/node-image-version` label on each NotReady node showed an Ubuntu version affected by the advisory. The nodes did not appear on the Azure Portal VMs page, so the nodes could not simply be rebooted there, and the team had no documented way to SSH into its Kubernetes nodes.

Symptom Service outageNode NotReady

Root cause

Root cause Breaking change in an upgrade

Component Node operating system image
Contributing factor Limited debug access
Trigger
Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

Detection

The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

Remediation

Action items

Lessons

Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Node operating system image among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.