Cases2 cases

1 to 2 of 2

1. IL-0249 Large training jobs on a cluster of over 10,000 GPUs slowed for tens of seconds to nearly ten hours without crashing

   Alibaba Group2024

   Symptoms

   On a shared production cluster of over 10,000 GPUs, large hybrid-parallel training jobs kept running but slowed down, sometimes for hours, without crashing. Among 27 jobs of 512 to 1024 GPUs each submitted in July 2024, 16 slowed this way for 72 minutes on average, and their job completion time grew by 34.59% on average.

   Throughput degradation

   Root cause

   Capacity exhaustion

   Component

   InfiniBand fabricGPU hardware and memory

   Contributing factor

   Overlapping failuresMonitoring blind spot

   Detection

   The authors found fail-slows with small probe training jobs that tracked their own performance at runtime, and by inspecting by hand a one-month trace of large training jobs.

   Remediation

   • Found fail-slows by hand and handled them as fail-stops with checkpoint-and-restart

   Action items

   • Detect fail-slows at runtime from iteration times and locate the slow GPUs and links with Falcon-Detect
   • Mitigate fail-slows with Falcon-Mitigate, escalating from cheap strategies to checkpoint-and-restart only if the slowdown persists

   Lessons

   Fail-slows are usually transient and come from computation, through slow GPUs or CPU contention, or from communication, mostly network congestion. Computation fail-slows are rarer and shorter and cost less, while cross-node communication fail-slows are more common, last longer and slow training more. As training grows, several performance problems are more likely to occur at once, and together they can slow training by more than 90%.

   View case
2. IL-0429 GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadable

   Tsinghua University2022

   Symptoms

   The GLM-130B pretraining run on 96 nodes, started on 2022-05-06 at 04:00, was interrupted again and again by single nodes that broke down, lost connection or reported missing GPUs, and each time training had to be restarted from a saved step. On 2022-05-30 a restart got stuck in data loading because of a Lustre file system error, and the 2.3T text corpora could no longer be read.

   Training job failureNode NotReadyGPU unavailableStorage disconnected or stale mountsThroughput degradation

   Trigger

   Individual nodes of the 96-node allocation failed during training, through breakdowns, a lost connection, missing GPUs, a CPU error and an NVLink error, and on 2022-05-30 the Lustre file system that held the training corpus failed.

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryShared network filesystem

   Contributing factor

   –

   Detection

   The Lustre error was found when a restart from global_step23200 stayed stuck in data loading.

   Remediation

   • Restarted the training from the last saved step after each node failure
   • Changed the checkpoint saving interval from 500 steps to 100 steps
   • Switched the job to another node pool and restarted from step 15600
   • Removed the suspected replacement node n30076 from step 16600, which did not change the speed
   • Copied the 2.3T text corpora from a backup disk back to the Lustre file system, which took a few days

   View case

By root cause2