Cases2 cases

1 to 2 of 2

1. IL-0426 A 64-GPU training run on Jean Zay hung after 3.5 hours when rank 0 stuck in uninterruptible sleep on a frozen filesystem

   Hugging Face2023

   Symptoms

   A 64-GPU training run (experiment tr_141, 8 nodes of the Jean Zay cluster) stopped making progress after about 3.5 hours: the main log stopped being written and the job hung without exiting. The c10d timeout of 30 minutes did not abort it, and the job stayed frozen indefinitely. The same hang had already happened on a different set of nodes, which first pointed to a software problem.

   Training job failureNCCL timeout

   Trigger

   The filesystem of the Jean Zay HPC system froze and stopped responding while rank 0 on the first node was writing to its log.

   Root cause

   Dependency failure

   Component

   Shared network filesystemPyTorch

   Contributing factor

   Limited debug access

   Detection

   The engineer, who happened to be around, noticed that the main training log had stopped being written.

   View case
2. IL-0428 Test runs of 100B-200B models on 320 to 384 A100 GPUs hung once inter-node traffic grew, and each hang took nodes down

   BigScience2022

   Symptoms

   On Jean Zay, test runs of large models across 320 to 384 A100 GPUs hung: random GPUs stopped responding and the whole job hung until it timed out. Every run of the hanging setup left a few nodes down until someone rebooted them, and at one point 13 of 52 nodes were down.

   Training job failureNCCL timeoutGPU unavailableNode NotReady

   Trigger

   Growing the test model past about 100B parameters over 20 to 40 nodes, which raised inter-node communication past the level the system handled, with CUDA launches left asynchronous.

   Root cause

   unknown

   Component

   NCCLPyTorch

   Contributing factor

   Limited debug access

   Remediation

   • Set CUDA_LAUNCH_BLOCKING=1, which removed the hang with little effect on throughput at this scale

   Lessons

   Python's trace module is very useful for debugging hanging processes because it logs the last call each process made before it stopped responding, but it should be started later, just before train_step, to avoid gigabytes of traces from the dataloader and setup code.

   View case

By root cause2