Cases2 cases

1 to 2 of 2

1. IL-0426 A 64-GPU training run on Jean Zay hung after 3.5 hours when rank 0 stuck in uninterruptible sleep on a frozen filesystem

   Hugging Face2023

   Symptoms

   A 64-GPU training run (experiment tr_141, 8 nodes of the Jean Zay cluster) stopped making progress after about 3.5 hours: the main log stopped being written and the job hung without exiting. The c10d timeout of 30 minutes did not abort it, and the job stayed frozen indefinitely. The same hang had already happened on a different set of nodes, which first pointed to a software problem.

   Training job failureNCCL timeout

   Trigger

   The filesystem of the Jean Zay HPC system froze and stopped responding while rank 0 on the first node was writing to its log.

   Root cause

   Dependency failure

   Component

   Shared network filesystemPyTorch

   Contributing factor

   Limited debug access

   Detection

   The engineer, who happened to be around, noticed that the main training log had stopped being written.

   View case
2. IL-0585 Training jobs on Microsoft's Philly GPU cluster failed hours into runs when checkpoints could not be written to HDFS

   Microsoft2019

   Symptoms

   On Philly, Microsoft's shared GPU cluster for deep learning training, jobs failed because they could not create a model checkpoint after a number of epochs had completed. Over a 75-day trace from October to December 2017 this "Model ckpt error" occurred 1,995 times in 948 jobs from 85 users.

   Training job failureStorage disconnected or stale mounts

   Trigger

   A job reached the point where it writes a model checkpoint while HDFS, the cluster's distributed storage, had a transient error or its name node was recovering.

   Root cause

   Dependency failure

   Component

   Shared network filesystem

   Contributing factor

   –

   Detection

   The job failed and the scheduler retried it. The authors attributed the failures to checkpoint errors afterwards with a rule-based classifier that matches failure signatures in the stdout and stderr logs of failed jobs.

   Lessons

   Infrastructure failures such as checkpoint errors are rare compared with user errors, but they appear after long runs and so dominate the time until failures are detected. The authors recommend that schedulers classify error messages from job logs at runtime and adapt their retry policies, for example continuing to retry network timeouts while no longer retrying incorrect inputs, to avoid wasteful re-execution.

   View case

By root cause1