IL-0426 Hugging Face · 2023-03-04ai-infra

A 64-GPU training run on Jean Zay hung after 3.5 hours when rank 0 stuck in uninterruptible sleep on a frozen filesystem

Hugging Face was running the m4 experiment tr_141 on 64 GPUs across 8 nodes of the Jean Zay HPC cluster (GPU 0 on the stuck node was an NVIDIA A100-SXM4-80GB), with PyTorch 1.12, accelerate and DeepSpeed, CUDA_LAUNCH_BLOCKING=1 and num_workers=2. After about 3.5 hours of training the job froze. Because c10d never aborted the job, the engineer could study it while it hung: the log showed gradient clipping as the last step, py-spy put the other ranks in a torch.distributed barrier, and the one process that did not answer was rank 0 on node jean-zay-iam06 (GPU 0, process 280058). strace and py-spy could not attach to it, ps showed it in uninterruptible sleep, and nvidia-smi and dmesg showed nothing wrong with the GPU.

The next day the Jean Zay system administrators announced that the filesystem had suffered a meltdown. That explained the hang: rank 0 had made a system call to write to its log, the call never returned because the filesystem was frozen, the process went into uninterruptible sleep, and the other ranks waited for it in the barrier. The c10d timeout did not fire, so the whole job stayed frozen instead of failing, and the stuck process could only be cleared by rebooting the node.

Severity: –
Scope: A 64-GPU training run on 8 Jean Zay nodes hung indefinitely; the stuck rank 0 process on jean-zay-iam06 could only be cleared by rebooting the node
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

A 64-GPU training run (experiment tr_141, 8 nodes of the Jean Zay cluster) stopped making progress after about 3.5 hours: the main log stopped being written and the job hung without exiting. The c10d timeout of 30 minutes did not abort it, and the job stayed frozen indefinitely. The same hang had already happened on a different set of nodes, which first pointed to a software problem.

The last lines of the main log were per-process messages such as "Before sync_gradients in _do_batch" and "After clip in _do_batch", so gradient clipping was the last step that ran. A py-spy dump of every process showed the ranks waiting in "barrier (torch/distributed/distributed_c10d.py:2784)" called from accelerate's "wait_for_everyone". Node jean-zay-iam06 returned one report fewer than the other nodes, and running py-spy on it directly failed with "Error: Failed to suspend process" and "Reason: EPERM: Operation not permitted".

On jean-zay-iam06, nvidia-smi showed seven GPUs at 100% utilization and GPU 0 at 0%, held by process 280058. Attaching strace to that process failed with "strace: attach: ptrace(PTRACE_SEIZE, 280058): Operation not permitted", and ps showed it in state "Dl" (uninterruptible sleep, usually IO) while the other seven training processes were "Rl". A process in this state could only be cleared by rebooting the node.

The GPU itself looked healthy. nvidia-smi -q for GPU 0 showed 28 C, ECC mode enabled with zero correctable and uncorrectable errors, and no remapped rows; its clocks were at 1155 MHz against 1410 MHz on the other seven GPUs, which the engineer thought might be normal for a GPU at 0% utilization that had slowed itself down. dmesg had nothing from NVRM or nvidia around the time of the hang, only failed kernel stacks from the attempts to attach to the sleeping process.

Symptom Training job failure NCCL timeout

Root cause

Root cause Dependency failure

Component Shared network filesystem PyTorch

Contributing factor Limited debug access

Trigger: The filesystem of the Jean Zay HPC system froze and stopped responding while rank 0 on the first node was writing to its log.

Detection

The engineer, who happened to be around, noticed that the main training log had stopped being written.

Remediation

–

Action items

–

Lessons

–

Sources

https://github.com/huggingface/m4-logs/blob/master/tr_141-hanging/tr_141_cm409xPMD01_scale_leap_of_faith_v5_num_workers_0-try3/README.md · github

Related cases

IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026
same root cause · shared PyTorch
IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMsLambda · 2026 · sev1
same root cause · shared Shared network filesystem
IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024
same root cause · shared PyTorch
IL-0585 Training jobs on Microsoft's Philly GPU cluster failed hours into runs when checkpoints could not be written to HDFSMicrosoft · 2019
same root cause · shared Shared network filesystem

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Shared network filesystem, PyTorch among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0426
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0426/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.