Cases4 cases

1 to 4 of 4

1. IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardware

   Meta2024

   Symptoms

   Over a 54-day snapshot of Llama 3 405B pre-training the training job was interrupted 466 times. 47 interruptions were planned: automated maintenance such as firmware upgrades, or operator actions such as configuration or dataset updates. The other 419 were unexpected. Because training is synchronous, a failure on a single GPU can force a restart of the whole job.

   Training job failureGPU unavailableNCCL timeout

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLPyTorch

   Contributing factor

   Single point of failure

   Detection

   The communication library's state is monitored and a stalled collective is timed out automatically; on an NCCLX watchdog or heartbeat timeout the PyTorch NCCL flight recorder dumps its tracing data, including collective metadata and stack traces.

   Remediation

   • handled almost every interruption through automation, with significant manual intervention only three times in 54 days

   Action items

   • Reduce job startup and checkpointing time
   • Use the PyTorch NCCL flight recorder to record every communication event and collective duration and to dump traces on NCCLX watchdog or heartbeat timeouts
   • Turn on heavier tracing and metadata collection in production through online configuration changes, without a code release or job restart
   • Give PyTorch access to NCCLX internal state so stalls are detected and localised faster, with a snapshot of the failing collective's finished and pending transfers

   Lessons

   Training on 16K GPUs has more failure modes than Meta's much larger CPU clusters, and because training is synchronous a single GPU failure can force a restart of the whole job. Meta raised effective training time by cutting job startup and checkpoint time and by building tools for fast diagnosis, and kept it above 90% while automated firmware and kernel maintenance caused at least one interruption a day.

   View case
2. IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

   Imbue2024

   Symptoms

   When Imbue brought up a cluster of 4,088 H100 GPUs on 511 hosts, about 10% of the machines failed to boot, mostly from physical problems: unconnected or miswired Ethernet cables, iDRAC hardware issues, broken power supplies, bad NVMe drives, missing internal wires, and network cards or GPUs that did not show up. In single-node GPU tests, Ubuntu server logs showed links between the GPUs and the PCIe buses or network cards reporting "limited width: x4 < x16". A few hosts had NVMe drives that locked up the whole machine when touched, drives that appeared in random order and got the OS installed on the wrong one, wrong temperature readings that kept fans at 100%, and CPUs limited to 2 GHz.

   Link flappingNCCL timeoutThroughput degradationGPU unavailableTraining job failure

   Trigger

   Bringing a new InfiniBand-connected H100 cluster into service and running the first multi-node training jobs on it.

   Root cause

   Dependency failure

   Component

   InfiniBand fabricNCCLGPU driverPyTorch

   Contributing factor

   Unverified assumptionMonitoring blind spotPartial rollout

   Detection

   Automated hardware checks during provisioning, UFM alerts on temperature, error rates and flapping ports, Prometheus node exporter and DCGM metrics, per-batch MFU monitoring, and live debugging of stalled processes with Py-Spy, GDB and NVIDIA Nsight.

   Remediation

   • Automated checks for broken machines and sent faulty ones back to Dell or to data center staff
   • Reseated GPUs and internal PCIe cables and applied firmware upgrades
   • Downgraded the NVIDIA driver that caused wrong temperature readings
   • Rewired the top tier of the InfiniBand fabric into a single network and added checks that the wiring matched the design
   • Cleaned and reseated alerting ports and disabled the remaining alerting transceivers
   • Ran a burn-in workload through every InfiniBand port and disabled the ports that did not survive it
   • Enabled GPUDirect RDMA with an extra kernel module and PCIe ACS disabled
   • Disabled automatic garbage collection and scheduled it at set intervals on all hosts

   Action items

   • Release host-level health checks that confirm a host is free from known errors
   • Release an NCCL patch that improves logging around errors and stalls
   • Release a GPU stress test and NVLink and InfiniBand networking tests
   • Release a script that parses the UFM event log and decides which network ports to disable
   • Release a script that generates a burn-in workload exercising every InfiniBand link
   • Relaunch crashed runs automatically from the last checkpoint on the machines that pass health checks
   • Replace the disabled alerting transceivers

   Lessons

   Keep 10 to 20% more machines than a run needs, and network them so any working subset can be used. On the newest hardware expect about 3% of machines to break each week, mostly the same few machines breaking in different ways, so growing a set of known reliable "golden" hosts works better than chasing random failures. Write tests and automation for every kind of failure met, because each one comes back, and write tools that make opaque errors readable. Change only one thing at a time, and double-check the claims of new tools and people before building on them.

   View case
3. IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted it

   BigScience2022

   Symptoms

   During BigScience's 176B-parameter multilingual training (tr11-176B-ml) on the Jean Zay cluster, launched on 2022-03-11, one of the GPUs died about once a week. Most of the time the SLURM job crashed and restarted automatically from the last checkpoint.

   Training job failureGPU unavailable

   Trigger

   A GPU on one of the training nodes failed during the run.

   Root cause

   Hardware failure

   Component

   NCCLPyTorchSlurm job scheduler

   Contributing factor

   Unreviewed defaultsLimited debug access

   Detection

   There was no automatic detection of a hung job; it was noticed only when someone watching the training saw that it was stuck.

   Remediation

   • raised checkpoint frequency to every 200 iterations and then to every 100 iterations (SAVE_INTERVAL=100)
   • set NCCL_ASYNC_ERROR_HANDLING=1 so NCCL problems crash the job instead of hanging it
   • added the srun options --wait=60 and --kill-on-bad-exit=1 after testing them on 2 nodes

   Lessons

   On a long run across hundreds of GPUs, a dead GPU every week is routine; what costs time is the case where a process crashes and the job hangs instead of exiting, because then nothing restarts it. Making NCCL errors fatal and having the scheduler end the whole step on a bad exit is meant to turn such a hang into a crash that the existing automatic restart handles; only the srun part was tested. Frequent checkpoints bound the loss per crash, at the price of save time for a 2.3TB checkpoint.

   View case
4. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hung

   Meta AI2021

   Symptoms

   Between the team's 17 November 2021 update and 3 December 2021, the 175B-parameter OPT training run went down and had to be restarted more than 40 times, for hardware, infrastructure and training stability reasons. A hardware fault could stop the run for hours at any time of day, and several did over the Thanksgiving break.

   Training job failureGPU unavailable

   Trigger

   Nodes in the cloud training cluster failed with ECC errors, and each failure took the training run down.

   Root cause

   Dependency failure

   Component

   GPU hardware and memoryNCCLInfiniBand fabricObject storage

   Contributing factor

   No headroomLimited debug accessNo runbook

   Detection

   During these two weeks the team added monitoring of train.log, GPU burn-in testing to detect memory errors, and automated InfiniBand testing.

   Remediation

   • replaced failed nodes, and later did it through a script
   • added GPU burn-in testing to detect memory errors
   • automated InfiniBand testing
   • added monitoring of train.log
   • debugged the cloud provider's Python API until checkpoint downloads from the blob store worked well enough

   Action items

   • Run an on-call rotation for the training run, with runbooks and tooling for the most common issues, and invite more people to join it

   Lessons

   At this scale cloud hardware failures are certain, and the team concluded that automating the handling of hardware failures is necessary. Without spare nodes and burn-in tests, each failure costs hours and a replacement can bring the same fault back. The team also concluded that cloud infrastructure instability is something to always prepare for, because the customer has no control over the underlying hardware. A checkpoint large enough to need hundreds of multi-gigabyte downloads makes the storage path part of recovery time.

   View case

By root cause2