Cases2 cases

1 to 2 of 2

1. IL-0428 Test runs of 100B-200B models on 320 to 384 A100 GPUs hung once inter-node traffic grew, and each hang took nodes down

   BigScience2022

   Symptoms

   On Jean Zay, test runs of large models across 320 to 384 A100 GPUs hung: random GPUs stopped responding and the whole job hung until it timed out. Every run of the hanging setup left a few nodes down until someone rebooted them, and at one point 13 of 52 nodes were down.

   Training job failureNCCL timeoutGPU unavailableNode NotReady

   Trigger

   Growing the test model past about 100B parameters over 20 to 40 nodes, which raised inter-node communication past the level the system handled, with CUDA launches left asynchronous.

   Root cause

   unknown

   Component

   NCCLPyTorch

   Contributing factor

   Limited debug access

   Remediation

   • Set CUDA_LAUNCH_BLOCKING=1, which removed the hang with little effect on throughput at this scale

   Lessons

   Python's trace module is very useful for debugging hanging processes because it logs the last call each process made before it stopped responding, but it should be started later, just before train_step, to avoid gigabytes of traces from the dataloader and setup code.

   View case
2. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC

   Binder Project2018

   Symptoms

   On 13 March 2018 at 07:28 CET the mybinder.org team noticed that JupyterHub had not been spawning new users and that the launch success rate on the Grafana dashboard had been at zero for several hours. Many pods were in the states "Unknown" and "NodeLost", on at least the nodes `10.128.0.7` and `10.128.0.5`.

   Service outageNode NotReadyPods stuck in TerminatingPods stuck in ContainerCreating

   Root cause

   unknown

   Component

   Persistent volumes and stateful setsNode pool

   Contributing factor

   Pages not delivered

   Detection

   The team noticed only after several hours, when the launch success rate on the Grafana dashboard had been at zero for a long time. No Stackdriver alert email arrived in time, because those emails were marked as spam and reached the team hours later.

   Remediation

   • Cordoned the bad nodes
   • Deleted the hub pod in Unknown state and the hub pod stuck in ContainerCreating
   • Reset node gke-prod-a-ssd-pool-32-134a959a-n2sk from the GCP web interface to force the PVC to be released
   • Force-deleted every pod in Unknown or NodeLost state with kubectl delete --grace-period=0 --force

   Action items

   • Investigate why the outage started, through Stackdriver logs and Google Cloud incidents
   • Set up an alerting service that does not depend on Stackdriver
   • Document how to check whether a PVC has not been released
   • Document how to restart a node by hand when commands are not working
   • Document how to delete pods that are in Unknown state

   Lessons

   The error went unnoticed for a long time because Stackdriver alert emails were marked as spam and arrived hours late. Once the problem was found, each step was logged and the steps were made more forceful as needed, so the hub was down only briefly after detection.

   View case

By root cause1