Incident Library

IL-0507 UC Berkeley · 2017-10-10kubernetesazure

Student pods on UC Berkeley DataHub got stuck in Terminating or would not start because Docker hung on two Azure nodes

On 10 October 2017 some DataHub user pods did not start or terminate correctly. All affected pods ran on two nodes, k8s-pool1-19522833-13 and k8s-pool2-19522833-9, where the Docker daemon had hung and did not respond, so pods on those nodes could neither start nor stop. On k8s-pool1-19522833-13 the hypercube process was at 766% CPU, while it was normal on the other node. The write-up concludes that Docker hung on the two nodes and does not say why; at 17:25 the team noticed that /var/lib/docker on k8s-pool1-19522833-13 was on the root disk (sda) rather than on /mnt (sdb).

The stuck pods were force-deleted with `kubectl --namespace=prod delete pod jupyter-<name> --grace-period=0 --force` and both nodes were cordoned, after which the students' servers started. k8s-pool1-19522833-13 came back with a responsive Docker only after it was stopped and then started from the Azure portal at 14:30, and it was uncordoned; k8s-pool2-19522833-9 was stopped from the Azure portal at 14:30 and started again there once it had finished stopping at 14:45. A third node, pool1-...-14, cordoned on an earlier day, was rebooted, came back online and was uncordoned.

Severity
–
Scope
UC Berkeley DataHub: student servers whose pods ran on two nodes could not start or stop
Users affected
students whose pods were placed on the two nodes
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On 10 October 2017 at 10:45 AM, a post on the course Piazza reported that two students could not start their servers on UC Berkeley's DataHub, and the /hub/admin interface could not start them either. At first it was thought the students might have run out of memory.

The students' pods were stuck in Terminating and did not respond to an explicit delete. After another report in the course Slack at 13:27, an operator saw that node k8s-pool1-19522833-13 also hosted many more pods stuck in Terminating and that `docker ps` hung on it; a check of the whole cluster for nodes with an unusual number of Terminating pods found k8s-pool2-19522833-9 in the same state. The two nodes had loads of 476 and 530.

On k8s-pool2-19522833-9, `systemctl status docker` showed errors such as "containerd: container did not start before the specified timeout", "Create container failed with error: containerd: container did not start before the specified timeout" and "write unix /var/run/docker.sock->@: write: broken pipe", and `ps aux | grep exe | wc -l` counted 520 matching processes, the first of them `/proc/self/exe init` in state Dsl. Rebooting k8s-pool1-19522833-13 from the shell, and then restarting it from the Azure portal, did not bring it back online.

Symptom Pods stuck in Terminating

Root cause

Root cause unknown

Component Container runtime
Contributing factor –
Trigger
–

Detection

Students reported on the course Piazza at 10:45 AM, and later in the course Slack, that their servers would not start.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Container runtime among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.