Cases2 cases

1 to 2 of 2

1. IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardware

   Meta2024

   Symptoms

   Over a 54-day snapshot of Llama 3 405B pre-training the training job was interrupted 466 times. 47 interruptions were planned: automated maintenance such as firmware upgrades, or operator actions such as configuration or dataset updates. The other 419 were unexpected. Because training is synchronous, a failure on a single GPU can force a restart of the whole job.

   Training job failureGPU unavailableNCCL timeout

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLPyTorch

   Contributing factor

   Single point of failure

   Detection

   The communication library's state is monitored and a stalled collective is timed out automatically; on an NCCLX watchdog or heartbeat timeout the PyTorch NCCL flight recorder dumps its tracing data, including collective metadata and stack traces.

   Remediation

   • handled almost every interruption through automation, with significant manual intervention only three times in 54 days

   Action items

   • Reduce job startup and checkpointing time
   • Use the PyTorch NCCL flight recorder to record every communication event and collective duration and to dump traces on NCCLX watchdog or heartbeat timeouts
   • Turn on heavier tracing and metadata collection in production through online configuration changes, without a code release or job restart
   • Give PyTorch access to NCCLX internal state so stalls are detected and localised faster, with a snapshot of the failing collective's finished and pending transfers

   Lessons

   Training on 16K GPUs has more failure modes than Meta's much larger CPU clusters, and because training is synchronous a single GPU failure can force a restart of the whole job. Meta raised effective training time by cutting job startup and checkpoint time and by building tools for fast diagnosis, and kept it above 90% while automated firmware and kernel maintenance caused at least one interruption a day.

   View case
2. IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted it

   BigScience2022

   Symptoms

   During BigScience's 176B-parameter multilingual training (tr11-176B-ml) on the Jean Zay cluster, launched on 2022-03-11, one of the GPUs died about once a week. Most of the time the SLURM job crashed and restarted automatically from the last checkpoint.

   Training job failureGPU unavailable

   Trigger

   A GPU on one of the training nodes failed during the run.

   Root cause

   Hardware failure

   Component

   NCCLPyTorchSlurm job scheduler

   Contributing factor

   Unreviewed defaultsLimited debug access

   Detection

   There was no automatic detection of a hung job; it was noticed only when someone watching the training saw that it was stuck.

   Remediation

   • raised checkpoint frequency to every 200 iterations and then to every 100 iterations (SAVE_INTERVAL=100)
   • set NCCL_ASYNC_ERROR_HANDLING=1 so NCCL problems crash the job instead of hanging it
   • added the srun options --wait=60 and --kill-on-bad-exit=1 after testing them on 2 nodes

   Lessons

   On a long run across hundreds of GPUs, a dead GPU every week is routine; what costs time is the case where a process crashes and the job hangs instead of exiting, because then nothing restarts it. Making NCCL errors fatal and having the scheduler end the whole step on a bad exit is meant to turn such a hang into a crash that the existing automatic restart handles; only the srun part was tested. Frequent checkpoints bound the loss per crash, at the price of save time for a 2.3TB checkpoint.

   View case

By root cause1