Cases2 cases

1 to 2 of 2

1. IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardware

   Meta2024

   Symptoms

   Over a 54-day snapshot of Llama 3 405B pre-training the training job was interrupted 466 times. 47 interruptions were planned: automated maintenance such as firmware upgrades, or operator actions such as configuration or dataset updates. The other 419 were unexpected. Because training is synchronous, a failure on a single GPU can force a restart of the whole job.

   Training job failureGPU unavailableNCCL timeout

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLPyTorch

   Contributing factor

   Single point of failure

   Detection

   The communication library's state is monitored and a stalled collective is timed out automatically; on an NCCLX watchdog or heartbeat timeout the PyTorch NCCL flight recorder dumps its tracing data, including collective metadata and stack traces.

   Remediation

   • handled almost every interruption through automation, with significant manual intervention only three times in 54 days

   Action items

   • Reduce job startup and checkpointing time
   • Use the PyTorch NCCL flight recorder to record every communication event and collective duration and to dump traces on NCCLX watchdog or heartbeat timeouts
   • Turn on heavier tracing and metadata collection in production through online configuration changes, without a code release or job restart
   • Give PyTorch access to NCCLX internal state so stalls are detected and localised faster, with a snapshot of the failing collective's finished and pending transfers

   Lessons

   Training on 16K GPUs has more failure modes than Meta's much larger CPU clusters, and because training is synchronous a single GPU failure can force a restart of the whole job. Meta raised effective training time by cutting job startup and checkpoint time and by building tools for fast diagnosis, and kept it above 90% while automated firmware and kernel maintenance caused at least one interruption a day.

   View case
2. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hung

   Meta AI2021

   Symptoms

   Between the team's 17 November 2021 update and 3 December 2021, the 175B-parameter OPT training run went down and had to be restarted more than 40 times, for hardware, infrastructure and training stability reasons. A hardware fault could stop the run for hours at any time of day, and several did over the Thanksgiving break.

   Training job failureGPU unavailable

   Trigger

   Nodes in the cloud training cluster failed with ECC errors, and each failure took the training run down.

   Root cause

   Dependency failure

   Component

   GPU hardware and memoryNCCLInfiniBand fabricObject storage

   Contributing factor

   No headroomLimited debug accessNo runbook

   Detection

   During these two weeks the team added monitoring of train.log, GPU burn-in testing to detect memory errors, and automated InfiniBand testing.

   Remediation

   • replaced failed nodes, and later did it through a script
   • added GPU burn-in testing to detect memory errors
   • automated InfiniBand testing
   • added monitoring of train.log
   • debugged the cloud provider's Python API until checkpoint downloads from the blob store worked well enough

   Action items

   • Run an on-call rotation for the training run, with runbooks and tooling for the most common issues, and invite more people to join it

   Lessons

   At this scale cloud hardware failures are certain, and the team concluded that automating the handling of hardware failures is necessary. Without spare nodes and burn-in tests, each failure costs hours and a replacement can bring the same fault back. The team also concluded that cloud infrastructure instability is something to always prepare for, because the customer has no control over the underlying hardware. A checkpoint large enough to need hundreds of multi-gigabyte downloads makes the storage path part of recovery time.

   View case

By root cause2