Cases6 cases

5 to 6 of 6

1. IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted it

   BigScience2022

   Symptoms

   During BigScience's 176B-parameter multilingual training (tr11-176B-ml) on the Jean Zay cluster, launched on 2022-03-11, one of the GPUs died about once a week. Most of the time the SLURM job crashed and restarted automatically from the last checkpoint.

   Training job failureGPU unavailable

   Trigger

   A GPU on one of the training nodes failed during the run.

   Root cause

   Hardware failure

   Component

   NCCLPyTorchSlurm job scheduler

   Contributing factor

   Unreviewed defaultsLimited debug access

   Detection

   There was no automatic detection of a hung job; it was noticed only when someone watching the training saw that it was stuck.

   Remediation

   • raised checkpoint frequency to every 200 iterations and then to every 100 iterations (SAVE_INTERVAL=100)
   • set NCCL_ASYNC_ERROR_HANDLING=1 so NCCL problems crash the job instead of hanging it
   • added the srun options --wait=60 and --kill-on-bad-exit=1 after testing them on 2 nodes

   Lessons

   On a long run across hundreds of GPUs, a dead GPU every week is routine; what costs time is the case where a process crashes and the job hangs instead of exiting, because then nothing restarts it. Making NCCL errors fatal and having the scheduler end the whole step on a bad exit is meant to turn such a hang into a crash that the existing automatic restart handles; only the srun part was tested. Frequent checkpoints bound the loss per crash, at the price of save time for a 2.3TB checkpoint.

   View case
2. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hung

   Meta AI2021

   Symptoms

   Between the team's 17 November 2021 update and 3 December 2021, the 175B-parameter OPT training run went down and had to be restarted more than 40 times, for hardware, infrastructure and training stability reasons. A hardware fault could stop the run for hours at any time of day, and several did over the Thanksgiving break.

   Training job failureGPU unavailable

   Trigger

   Nodes in the cloud training cluster failed with ECC errors, and each failure took the training run down.

   Root cause

   Dependency failure

   Component

   GPU hardware and memoryNCCLInfiniBand fabricObject storage

   Contributing factor

   No headroomLimited debug accessNo runbook

   Detection

   During these two weeks the team added monitoring of train.log, GPU burn-in testing to detect memory errors, and automated InfiniBand testing.

   Remediation

   • replaced failed nodes, and later did it through a script
   • added GPU burn-in testing to detect memory errors
   • automated InfiniBand testing
   • added monitoring of train.log
   • debugged the cloud provider's Python API until checkpoint downloads from the blob store worked well enough

   Action items

   • Run an on-call rotation for the training run, with runbooks and tooling for the most common issues, and invite more people to join it

   Lessons

   At this scale cloud hardware failures are certain, and the team concluded that automating the handling of hardware failures is necessary. Without spare nodes and burn-in tests, each failure costs hours and a replacement can bring the same fault back. The team also concluded that cloud infrastructure instability is something to always prepare for, because the customer has no control over the underlying hardware. A checkpoint large enough to need hundreds of multi-gigabyte downloads makes the storage path part of recovery time.

   View case

By root cause3