Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0426 A 64-GPU training run on Jean Zay hung after 3.5 hours when rank 0 stuck in uninterruptible sleep on a frozen filesystem
    Hugging Face2023
    Symptoms

    A 64-GPU training run (experiment tr_141, 8 nodes of the Jean Zay cluster) stopped making progress after about 3.5 hours: the main log stopped being written and the job hung without exiting. The c10d timeout of 30 minutes did not abort it, and the job stayed frozen indefinitely. The same hang had already happened on a different set of nodes, which first pointed to a software problem.

    Training job failureNCCL timeout
    Trigger

    The filesystem of the Jean Zay HPC system froze and stopped responding while rank 0 on the first node was writing to its log.

    Root cause
    Dependency failure
    Component
    Shared network filesystemPyTorch
    Contributing factor
    Limited debug access
    Detection

    The engineer, who happened to be around, noticed that the main training log had stopped being written.

    View case
Check your cluster for this

This case involves PyTorch and Shared network filesystem. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
PyTorchShared network filesystem
Contributing factor
Limited debug access
Organisation
Hugging Face
  1. IL-0426 A 64-GPU training run on Jean Zay hung after 3.5 hours when rank 0 stuck in uninterruptible sleep on a frozen filesystem
    Hugging Face2023