IL-0421 Alibaba Group · 2025-05-23 (published)ai-infra
Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023
The paper by Alibaba engineers describes C4, a system deployed in Alibaba's production AI training clusters, whose nodes have 8 NVIDIA H800 GPUs and 8 BlueField-3 NICs with two bonded 200Gbps ports each on a 3-tier Clos fat-tree network, running Megatron-LM and DeepSpeed. Training runs in bulk synchronous mode, so an error in any worker blocks the others and crashes the whole job, which then restarts from its last checkpoint; the paper states that high-end GPUs show high error rates, and that without efficient detection and diagnosis tools about 30% of the team's time went into error detection, diagnosis, isolating defective nodes and restarting, leaving 70% for computing. The measurements come from the crash causes of one 4,096-GPU job over one month and from historical job logs of an internal customer's 2400-GPU job before (June 2023) and after (December 2023) C4's fault detection was deployed.
In June 2023, GPU-related errors (ECC errors, NVLink errors and CUDA errors) caused 12.53% of that job's total downtime, about two thirds of the diagnosis and isolation overhead, in which ECC/NVLink errors took 8.34%, CUDA errors 4.19%, CCL timeouts 3%, ACK timeouts 1.8% and unknown causes 2.29%. Post-checkpoint loss was high because users had set infrequent checkpoints, not expecting high error rates in a large cluster. Alibaba allocates 64 backup GPUs on 8 servers for every 1024 GPUs on 128 servers, and improved cooling with higher fan speeds and better air conditioning. C4D, the detection part of C4, extends Alibaba's collective communication library ACCL to record communicator, operation and transport-layer timings; a central master compares workers at synchronisation points to find hangs and slow communication or computation, tells the job steering service to isolate the faulty nodes and restart from the latest valid checkpoint, and hands the event to offline root cause analysis.
By December 2023 the job's error-induced downtime had fallen to 1.16%: diagnosis and isolation to 0.73%, post-checkpoint loss to 0.23% after users began checkpointing every 10 minutes, detection to 0.05% and re-initialisation to 0.15%. C4D found faults within tens of seconds, though isolating nodes and restarting still took minutes. GPU-related errors became 3.2 times less frequent and the time they cost fell 41.8 times, other errors fell 3.4 times in frequency and 16.5 times in time, and the average error rate fell 3.33 times after the most vulnerable components were identified and enhanced over the six months. C4D has limited ability to detect errors during job initialisation, before collective communication starts.
- Severity
- –
- Scope
- LLM training jobs crashed and restarted from checkpoints; one 2400-GPU job lost 31.19% of its time to errors in June 2023, and one 4,096-GPU job crashed 40 times in a month
- Users affected
- Alibaba's customers training LLMs, including an internal customer
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
Large LLM training jobs on Alibaba's AI clusters crashed again and again and had to restart from the last valid checkpoint. One of the largest jobs, on 4,096 GPUs in a newly deployed cluster, crashed 40 times in one month, and most crashes reached users only as "NCCL Errors".
The error users saw rarely named the cause. When a hardware problem in one GPU crashed one worker process, its peers got no acknowledgment for the messages they had sent to it and the communication library raised error code 12, the same code that network disconnections, container evictions after memory overflows or user code errors such as tensor size mismatches also produce. Behind the 40 crashes were ECC/NVLink errors (27.5%), ACK timeouts (27.5%), NCCL timeouts (20%), CUDA errors (12.5%) and other causes shown to users as a network error (12.5%); about 82.5% of the errors were confined to specific nodes or single devices.
The time went into finding the bad node rather than the crash itself. Identifying the cause and locating the defective node by hand took hours or even days in a cluster with thousands of GPUs. Users were also not told right away that a job had stalled: a PyTorch job could hang for up to 30 minutes until the elastic agent killed its processes. In June 2023, a 2400-GPU job training a 175 billion parameter GPT model lost 31.19% of its time to errors: 19.65% to diagnosis and isolation, 7.53% to work done after the last checkpoint, 3.41% to detecting the crash and 0.6% to re-initialisation.
Root cause
Root cause Hardware failure
- Trigger
- –
Detection
Before C4D, users found crashes through NCCL errors and could stay unaware of a stalled job for up to 30 minutes until the PyTorch elastic agent killed it; C4D later detects hangs and slowdowns from collective communication timings within tens of seconds.
Remediation
- Allocated 64 backup GPUs on 8 servers for every 1024 GPUs on 128 servers, so that parallel training on any 128 servers of the 136-server pool keeps the same communication and performance
- Deployed C4D to detect faulty nodes automatically, isolate them and restart the job from the most recent valid checkpoint
- Saved checkpoints more often, every 10 minutes, to cut the work lost after each crash
- Enhanced cooling with higher fan speeds and better air conditioning to keep GPUs from overheating
- Identified and enhanced the most vulnerable components of the system, lowering the average error rate 3.33 times
Action items
- Take load variation into account in C4D for more precise slow detection
Lessons
Most crash causes are confined to specific nodes or devices, so isolating the faulty component quickly and restarting, while leaving root cause analysis for offline work, removes most of the error-induced downtime. Because synchronous training makes many different faults look like the same communication error, diagnosis has to use the regular rhythm of the workers at synchronisation points rather than the error message, and frequent checkpoints matter because error rates in large clusters are high.
Sources
References
Related cases
IL-0127Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024same root cause · shared GPU hardware and memory, NCCL, PyTorch
IL-0239LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024same root cause · shared GPU hardware and memory, NCCL
IL-0125A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022same root cause · shared NCCL, PyTorch
IL-0416Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026same root cause · shared GPU hardware and memory
IL-0478A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training stepsAmazon · 2026same root cause · shared NCCL
IL-0415GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025same root cause · shared GPU hardware and memory
IL-0482Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a dayNERSC · 2025same root cause · shared GPU hardware and memory
IL-0414Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025same root cause · shared GPU hardware and memory
IL-0411Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checksMeta · 2024same root cause · shared GPU hardware and memory
IL-0238Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024same root cause · shared GPU hardware and memory
IL-0429GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022same root cause · shared GPU hardware and memory
IL-0427A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed tooBigScience · 2021same root cause · shared GPU hardware and memory
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0421
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0421/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.