Cases1 case

1 to 1 of 1

1. IL-0585 Training jobs on Microsoft's Philly GPU cluster failed hours into runs when checkpoints could not be written to HDFS

   Microsoft2019

   Symptoms

   On Philly, Microsoft's shared GPU cluster for deep learning training, jobs failed because they could not create a model checkpoint after a number of epochs had completed. Over a 75-day trace from October to December 2017 this "Model ckpt error" occurred 1,995 times in 948 jobs from 85 users.

   Training job failureStorage disconnected or stale mounts

   Trigger

   A job reached the point where it writes a model checkpoint while HDFS, the cluster's distributed storage, had a transient error or its name node was recovering.

   Root cause

   Dependency failure

   Component

   Shared network filesystem

   Contributing factor

   –

   Detection

   The job failed and the scheduler retried it. The authors attributed the failures to checkpoint errors afterwards with a rule-based classifier that matches failure signatures in the stdout and stderr logs of failed jobs.

   Lessons

   Infrastructure failures such as checkpoint errors are rare compared with user errors, but they appear after long runs and so dominate the time until failures are detected. The authors recommend that schedulers classify error messages from job logs at runtime and adapt their retry policies, for example continuing to retry network timeouts while no longer retrying incorrect inputs, to avoid wasteful re-execution.

   View case

By root cause1