Cases2 cases

1 to 2 of 2

1. IL-0421 Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023

   Alibaba Group2025

   Symptoms

   Large LLM training jobs on Alibaba's AI clusters crashed again and again and had to restart from the last valid checkpoint. One of the largest jobs, on 4,096 GPUs in a newly deployed cluster, crashed 40 times in one month, and most crashes reached users only as "NCCL Errors".

   Training job failureNCCL timeoutGPU unavailable

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLPyTorch

   Contributing factor

   Monitoring blind spotUnverified assumption

   Detection

   Before C4D, users found crashes through NCCL errors and could stay unaware of a stalled job for up to 30 minutes until the PyTorch elastic agent killed it; C4D later detects hangs and slowdowns from collective communication timings within tens of seconds.

   Remediation

   • Allocated 64 backup GPUs on 8 servers for every 1024 GPUs on 128 servers, so that parallel training on any 128 servers of the 136-server pool keeps the same communication and performance
   • Deployed C4D to detect faulty nodes automatically, isolate them and restart the job from the most recent valid checkpoint
   • Saved checkpoints more often, every 10 minutes, to cut the work lost after each crash
   • Enhanced cooling with higher fan speeds and better air conditioning to keep GPUs from overheating
   • Identified and enhanced the most vulnerable components of the system, lowering the average error rate 3.33 times

   Action items

   • Take load variation into account in C4D for more precise slow detection

   Lessons

   Most crash causes are confined to specific nodes or devices, so isolating the faulty component quickly and restarting, while leaving root cause analysis for offline work, removes most of the error-induced downtime. Because synchronous training makes many different faults look like the same communication error, diagnosis has to use the regular rhythm of the workers at synchronisation points rather than the error message, and frequent checkpoints matter because error rates in large clusters are high.

   View case
2. IL-0427 A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed too

   BigScience2021

   Symptoms

   After many hours of training, one job of the tr1-13B Slurm job array crashed at iteration 43680 of 311541 during the backward pass. The next job in the array started by itself and crashed right away, before training began.

   Training job failureGPU unavailable

   Trigger

   One GPU on one node of the allocation developed a hardware problem that produced uncorrectable ECC errors.

   Root cause

   Hardware failure

   Component

   GPU hardware and memorySlurm job scheduler

   Contributing factor

   Automation amplificationMonitoring blind spot

   Detection

   Nobody was around to notice the first crash, and the Slurm scheduler started the next job in the array on its own, which crashed too.

   Remediation

   • Contacted the JZ administrators, who confirmed one of the nodes was faulty
   • Continued training with the next job, which was not placed on the faulty node

   Action items

   • Add %N to the Slurm output setting (#SBATCH --output=%x-%j-%N.out) so each node writes its own log file and the node with a corrupt GPU can be identified
   • Exclude a node known to be faulty from the allocation with sbatch --exclude

   Lessons

   Without a way to link an exception to the hostname of the node that raised it, the scheduler can keep placing the training on the faulty node and the training keeps crashing; per-node log files make the faulty node visible so it can be excluded.

   View case

By root cause1