Cases2 cases

1 to 2 of 2

1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node

   Fyra Labs2023

   Symptoms

   At around 2 AM ICT on 2023-04-09 the team saw unusual CPU use on one cluster node, which throttled every other service on it and made them fail. They presumed an upstream container runtime bug in containerd and runc and decided to fully reboot the affected nodes, taking the cluster down for maintenance on the assumption that it was not in active use.

   Sustained high node CPUService outage

   Trigger

   A full cold reboot of the nodes that showed unusual CPU use, followed by upgrades and cold reboots of the other nodes.

   Root cause

   unknown

   Component

   Multus meta-CNICalicoContainer runtime

   Contributing factor

   Harmful manual actionUnverified assumption

   Detection

   The team saw unusual CPU use on a node around 2 AM ICT and failing services on the rebooted nodes around 2:30 ICT; developers reported the Terra repositories down at 6:31 ICT.

   Remediation

   • uninstalled Multus and Calico and replaced them with the Cilium CNI
   • rolled back most cluster settings except the CNI

   Lessons

   Rebooting nodes to clear one problem can expose another; here a CNI chain could not reach the API server after a cold start. Repeating the same reboot on more nodes while the first failure was not understood spread the outage from a few nodes to the whole cluster. The team also found that Cilium's real resource use in their cluster was acceptable, contrary to the benchmark they had relied on.

   View case
2. IL-0507 Student pods on UC Berkeley DataHub got stuck in Terminating or would not start because Docker hung on two Azure nodes

   UC Berkeley2017

   Symptoms

   On 10 October 2017 at 10:45 AM, a post on the course Piazza reported that two students could not start their servers on UC Berkeley's DataHub, and the /hub/admin interface could not start them either. At first it was thought the students might have run out of memory.

   Pods stuck in Terminating

   Root cause

   unknown

   Component

   Container runtime

   Contributing factor

   –

   Detection

   Students reported on the course Piazza at 10:45 AM, and later in the course Slack, that their servers would not start.

   Remediation

   • Force-deleted the stuck user pods with kubectl --namespace=prod delete pod jupyter-<name> --grace-period=0 --force
   • Cordoned node k8s-pool1-19522833-13
   • Force-deleted the stuck pods on k8s-pool2-19522833-9 and cordoned that node
   • Stopped and started k8s-pool1-19522833-13 from the Azure portal, after which Docker responded, and uncordoned it
   • Stopped and started k8s-pool2-19522833-9 from the Azure portal

   Action items

   • When several student servers fail to start or stop, check whether their pods ran on the same nodes
   • Find out how many nodes do not mount /var/lib/docker on sdb1
   • Watch for elevated counts of pods stuck in Terminating

   View case

By root cause1