IL-0427 BigScience · 2021-08-12ai-infra
A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed too
BigScience trained the tr1-13B model with Megatron-DeepSpeed on the JZ cluster, running it as a chain of Slurm jobs in a job array so that each job started as soon as the previous one ended. One job crashed after many hours of training with an uncorrectable ECC error from CUDA. Nobody was around to notice, and the Slurm scheduler started the next job in the array, which was given the same node and crashed immediately during startup with a CUDA out-of-memory error at a torch.distributed barrier.
The team concluded that one of the GPUs had a hardware fault and contacted the JZ administrators, who confirmed that one of the nodes was faulty. The next training job did not land on that node and training continued. At that point the team had no way to match an exception to the hostname of the node that raised it, so a faulty node could keep being allocated and keep crashing the training. As a workaround the team added %N to the Slurm output file name, which gives every node its own log file and shows which node has the corrupt GPU, so the node can then be left out with sbatch --exclude.
- Severity
- –
- Scope
- The tr1-13B training run crashed twice in a row on the same faulty node and stopped until the next job landed on other nodes
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
After many hours of training, one job of the tr1-13B Slurm job array crashed at iteration 43680 of 311541 during the backward pass. The next job in the array started by itself and crashed right away, before training began.
The first crash ended in "RuntimeError: transform: failed to synchronize: cudaErrorECCUncorrectable: uncorrectable ECC error encountered", followed by "terminate called after throwing an instance of 'c10::Error'" and "what(): CUDA error: uncorrectable ECC error encountered", raised from torch.autograd.backward inside DeepSpeed's pipeline engine.
The second job, started at 2021-08-12 08:19:28 by the Slurm scheduler, initialized tensor model parallel size 2 and pipeline model parallel size 4, then failed while compiling and loading fused kernels: torch.distributed.barrier() in _compile_dependencies raised "RuntimeError: CUDA error: out of memory".
Nobody was watching when the first crash happened, and the logs gave no way to tie an exception to the hostname of the node it came from, so the faulty node could not be picked out and excluded.
Root cause
Root cause Hardware failure
- Trigger
- One GPU on one node of the allocation developed a hardware problem that produced uncorrectable ECC errors.
Detection
Nobody was around to notice the first crash, and the Slurm scheduler started the next job in the array on its own, which crashed too.
Remediation
- Contacted the JZ administrators, who confirmed one of the nodes was faulty
- Continued training with the next job, which was not placed on the faulty node
Action items
- Add %N to the Slurm output setting (#SBATCH --output=%x-%j-%N.out) so each node writes its own log file and the node with a corrupt GPU can be identified
- Exclude a node known to be faulty from the allocation with sbatch --exclude
Lessons
Without a way to link an exception to the hostname of the node that raised it, the scheduler can keep placing the training on the faulty node and the training keeps crashing; per-node log files make the faulty node visible so it can be excluded.
Sources
Related cases
IL-0411Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checksMeta · 2024same root cause · shared GPU hardware and memory, Slurm job scheduler
IL-0416Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026same root cause · shared GPU hardware and memory
IL-0415GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025same root cause · shared GPU hardware and memory
IL-0482Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a dayNERSC · 2025same root cause · shared GPU hardware and memory
IL-0421Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025same root cause · shared GPU hardware and memory
IL-0414Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025same root cause · shared GPU hardware and memory
IL-0238Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024same root cause · shared GPU hardware and memory
IL-0127Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024same root cause · shared GPU hardware and memory
IL-0239LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024same root cause · shared GPU hardware and memory
IL-0125A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022same root cause · shared Slurm job scheduler
IL-0429GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022same root cause · shared GPU hardware and memory
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0427
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0427/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.