Cases1 case

1 to 1 of 1

1. IL-0428 Test runs of 100B-200B models on 320 to 384 A100 GPUs hung once inter-node traffic grew, and each hang took nodes down

   BigScience2022

   Symptoms

   On Jean Zay, test runs of large models across 320 to 384 A100 GPUs hung: random GPUs stopped responding and the whole job hung until it timed out. Every run of the hanging setup left a few nodes down until someone rebooted them, and at one point 13 of 52 nodes were down.

   Training job failureNCCL timeoutGPU unavailableNode NotReady

   Trigger

   Growing the test model past about 100B parameters over 20 to 40 nodes, which raised inter-node communication past the level the system handled, with CUDA launches left asynchronous.

   Root cause

   unknown

   Component

   NCCLPyTorch

   Contributing factor

   Limited debug access

   Remediation

   • Set CUDA_LAUNCH_BLOCKING=1, which removed the hang with little effect on throughput at this scale

   Lessons

   Python's trace module is very useful for debugging hanging processes because it logs the last call each process made before it stopped responding, but it should be started later, just before train_step, to avoid gigabytes of traces from the dataloader and setup code.

   View case

By root cause1