Cases3 cases

1 to 3 of 3

1. IL-0426 A 64-GPU training run on Jean Zay hung after 3.5 hours when rank 0 stuck in uninterruptible sleep on a frozen filesystem

   Hugging Face2023

   Symptoms

   A 64-GPU training run (experiment tr_141, 8 nodes of the Jean Zay cluster) stopped making progress after about 3.5 hours: the main log stopped being written and the job hung without exiting. The c10d timeout of 30 minutes did not abort it, and the job stayed frozen indefinitely. The same hang had already happened on a different set of nodes, which first pointed to a software problem.

   Training job failureNCCL timeout

   Trigger

   The filesystem of the Jean Zay HPC system froze and stopped responding while rank 0 on the first node was writing to its log.

   Root cause

   Dependency failure

   Component

   Shared network filesystemPyTorch

   Contributing factor

   Limited debug access

   Detection

   The engineer, who happened to be around, noticed that the main training log had stopped being written.

   View case
2. IL-0429 GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadable

   Tsinghua University2022

   Symptoms

   The GLM-130B pretraining run on 96 nodes, started on 2022-05-06 at 04:00, was interrupted again and again by single nodes that broke down, lost connection or reported missing GPUs, and each time training had to be restarted from a saved step. On 2022-05-30 a restart got stuck in data loading because of a Lustre file system error, and the 2.3T text corpora could no longer be read.

   Training job failureNode NotReadyGPU unavailableStorage disconnected or stale mountsThroughput degradation

   Trigger

   Individual nodes of the 96-node allocation failed during training, through breakdowns, a lost connection, missing GPUs, a CPU error and an NVLink error, and on 2022-05-30 the Lustre file system that held the training corpus failed.

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryShared network filesystem

   Contributing factor

   –

   Detection

   The Lustre error was found when a restart from global_step23200 stayed stuck in data loading.

   Remediation

   • Restarted the training from the last saved step after each node failure
   • Changed the checkpoint saving interval from 500 steps to 100 steps
   • Switched the job to another node pool and restarted from step 15600
   • Removed the suspected replacement node n30076 from step 16600, which did not change the speed
   • Copied the 2.3T text corpora from a backup disk back to the Lustre file system, which took a few days

   View case
3. IL-0585 Training jobs on Microsoft's Philly GPU cluster failed hours into runs when checkpoints could not be written to HDFS

   Microsoft2019

   Symptoms

   On Philly, Microsoft's shared GPU cluster for deep learning training, jobs failed because they could not create a model checkpoint after a number of epochs had completed. Over a 75-day trace from October to December 2017 this "Model ckpt error" occurred 1,995 times in 948 jobs from 85 users.

   Training job failureStorage disconnected or stale mounts

   Trigger

   A job reached the point where it writes a model checkpoint while HDFS, the cluster's distributed storage, had a transient error or its name node was recovering.

   Root cause

   Dependency failure

   Component

   Shared network filesystem

   Contributing factor

   –

   Detection

   The job failed and the scheduler retried it. The authors attributed the failures to checkpoint errors afterwards with a rule-based classifier that matches failure signatures in the stdout and stderr logs of failed jobs.

   Lessons

   Infrastructure failures such as checkpoint errors are rare compared with user errors, but they appear after long runs and so dominate the time until failures are detected. The authors recommend that schedulers classify error messages from job logs at runtime and adapt their retry policies, for example continuing to retry network timeouts while no longer retrying incorrect inputs, to avoid wasteful re-execution.

   View case

By root cause2