Incident Library

IL-0585 Microsoft · 2019-08-08 (published)ai-infra

Training jobs on Microsoft's Philly GPU cluster failed hours into runs when checkpoints could not be written to HDFS

Philly is a Microsoft service that schedules and manages deep learning training jobs on large shared GPU clusters. The cluster in the study had hundreds of machines with thousands of GPUs of one model, in 2-GPU and 8-GPU servers, used HDFS as its distributed storage and a resource manager based on Apache YARN. Over 75 days from October to December 2017 it ran 96,260 jobs in 14 virtual clusters. The authors classified failed jobs with more than 230 rules that match failure signatures in each job's stdout and stderr logs.

One of the infrastructure failure reasons was the model checkpoint error: a job could not create its model checkpoint after a number of epochs, usually because of a transient error in HDFS or because the HDFS name node was recovering. It caused 1,995 failure events in 948 jobs. Because these jobs had usually run for hours, the checkpoint errors accounted for 21.73% of the summed runtime to failure of all failures, and 16.33% of runtime to failure multiplied by the number of GPUs. Jobs that failed were retried a fixed number of times before being marked unsuccessful.

Severity
–
Scope
Deep learning training jobs on one Philly GPU cluster from October to December 2017: 1,995 failure events in 948 jobs, 16.33% of the GPU time lost to failures
Users affected
85 users
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On Philly, Microsoft's shared GPU cluster for deep learning training, jobs failed because they could not create a model checkpoint after a number of epochs had completed. Over a 75-day trace from October to December 2017 this "Model ckpt error" occurred 1,995 times in 948 jobs from 85 users.

These failures came late in a job's life. The median runtime before the failure was 181.67 minutes, the 90th percentile 3,728.93 minutes and the 95th percentile 8,196.02 minutes, while failures from user syntax, semantic and configuration errors usually came within about 10 minutes. Model checkpoint errors and MPI runtime failures, the two infrastructure failure reasons the authors examined, together were only 6.2% of all failure events, but because they appeared after long runs they dominated the time until failures were detected.

The failures were spread over job sizes: 743 of the failure events were in 1-GPU jobs, 384 in jobs with 2 to 4 GPUs and 868 in jobs with more than 4 GPUs. Philly retries a failed job a fixed number of times and marks it unsuccessful when it keeps failing. The GPU time lost to a failure grows with both the runtime before the failure and the number of GPUs the job holds.

Symptom Training job failureStorage disconnected or stale mounts

Root cause

Root cause Dependency failure

Component Shared network filesystem
Contributing factor –
Trigger
A job reached the point where it writes a model checkpoint while HDFS, the cluster's distributed storage, had a transient error or its name node was recovering.

Detection

The job failed and the scheduler retried it. The authors attributed the failures to checkpoint errors afterwards with a rule-based classifier that matches failure signatures in the stdout and stderr logs of failed jobs.

Remediation

–

Action items

–

Lessons

Infrastructure failures such as checkpoint errors are rare compared with user errors, but they appear after long runs and so dominate the time until failures are detected. The authors recommend that schedulers classify error messages from job logs at runtime and adapt their retry policies, for example continuing to retry network timeouts while no longer retrying incorrect inputs, to avoid wasteful re-execution.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Shared network filesystem among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.