Incident Library

IL-0249 Alibaba Group · 2024-10-16 (published)ai-infranetworking

Large training jobs on a cluster of over 10,000 GPUs slowed for tens of seconds to nearly ten hours without crashing

The production cluster studied has more than 4,000 nodes and over 10,000 GPUs, about 1,800 NVIDIA H800 and 2,600 A100, on a spine-leaf RoCE network with up to 4×200/400 Gbps for A100/H800 nodes and NVSwitch inside each node. It runs large training jobs on more than 1,000 GPUs, online and offline inference, recommendation model jobs and short spot jobs for debugging. Hybrid-parallel training synchronises at every iteration, so a single slow GPU or link slows the whole job. These failures, called fail-slows or stragglers, do not stop the job the way fail-stops such as GPU hangs and runtime crashes do. Because production workloads could not be instrumented, the authors measured fail-slows with small probe training jobs, submitted repeatedly as spot workloads that landed on random nodes next to production jobs, and by inspecting by hand a one-month trace of jobs that used at least 512 GPUs.

Computation fail-slows were rare and short: 6 of 392 single-node jobs, 4 from CPU contention and 2 from GPU degradation, lasting about 10 minutes and adding 11.79% to job completion time. Communication fail-slows, mostly network congestion on a link, were common and longer: 43 of 107 four-node jobs, 42 from network congestion and 1 from CPU contention, lasting about 24 minutes and adding 15.45%. Among the 27 large jobs from July 2024, 16 slowed down, 13 from network congestion and 3 from network and GPU problems together, for 72 minutes on average and 34.59% longer job completion time; there was no CPU contention because those jobs ran alone on their nodes. Across all of them, fail-slows delayed the average job completion time by 1.34×.

The usual practice was to find fail-slows by hand and handle them like fail-stops with checkpoint-and-restart, which takes much time and work; dumping a GPT2-100B checkpoint takes nearly 100 minutes, longer than the mean fail-slow in this cluster. The authors built Falcon. Falcon-Detect hooks NCCL calls through LD_PRELOAD to infer each worker's iteration time, flags slow iterations with Bayesian online change-point detection plus a check that discards changes under 10%, profiles communication groups to find suspicious ones, and briefly pauses training to run GEMM and peer-to-peer link tests. Falcon-Mitigate escalates from doing nothing to redistributing micro-batches, then adjusting the parallel topology, and as a last resort checkpoint-and-restart. Falcon-Detect diagnosed 498 of 499 probe jobs correctly, and on a 64-GPU job with injected fail-slows Falcon reduced the slowdown by 60.1%.

Severity
–
Scope
Training jobs on a shared production GPU cluster: slower iterations and longer job completion time, more than 50% longer for 20% of the large jobs inspected
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On a shared production cluster of over 10,000 GPUs, large hybrid-parallel training jobs kept running but slowed down, sometimes for hours, without crashing. Among 27 jobs of 512 to 1024 GPUs each submitted in July 2024, 16 slowed this way for 72 minutes on average, and their job completion time grew by 34.59% on average.

Slow computation showed as falling training throughput together with falling GPU SM utilization. In one single-node probe job, throughput dropped twice, at 22 and 55 minutes, by up to 21.6%, and SM utilization fell on all four GPUs at once; a matrix computation test run during the slowdown found no GPU degradation, and the cause was a surge of high-CPU jobs on the same node that lowered the training job's CPU satisfaction rate. In another job, all four GPUs were under-utilized for the first 10 minutes because GPU0 ran 20% slower than the others at an unusually high temperature of nearly 70°C.

Slow communication was more common and lasted longer. In one four-node probe job, throughput fell from 0.57 to 0.41 iterations/s at t=90 minutes and to 0.31 iterations/s at t=265, and SM utilization dropped on all 8 GPUs at once although the GPUs were healthy; the slowdown matched a surge of congestion notification packets (CNPs) reported by the NICs. Inter-node RDMA was the least stable path, with a latency coefficient of variation of 0.29, against 0.09 for the PCIe switch and 0.02 or less for NVLink/NVSwitch and copies inside a GPU.

At scale the causes compounded. In one 1024-GPU job, network congestion at t=62 minutes cut throughput by 80%, GPU thermal throttling at about t=80 while the congestion continued brought throughput down to 10% of normal, and from t=120 another congestion lasting about two hours cut it by 85%. In the one-month trace of large jobs, 20% of the jobs were delayed by more than 50%, and across the study the slowdowns lasted from tens of seconds to nearly ten hours.

Symptom Throughput degradation

Root cause

Root cause Capacity exhaustion

Component InfiniBand fabricGPU hardware and memory
Contributing factor Overlapping failuresMonitoring blind spot
Trigger
–

Detection

The authors found fail-slows with small probe training jobs that tracked their own performance at runtime, and by inspecting by hand a one-month trace of large training jobs.

Remediation

Action items

Lessons

Fail-slows are usually transient and come from computation, through slow GPUs or CPU contention, or from communication, mostly network congestion. Computation fail-slows are rarer and shorter and cost less, while cross-node communication fail-slows are more common, last longer and slow training more. As training grows, several performance problems are more likely to occur at once, and together they can slow training by more than 90%.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, InfiniBand fabric, GPU hardware and memory among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.