Cases9 cases

1 to 5 of 9

1. IL-0416 Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errors

   ByteDance2026

   Symptoms

   In ByteDance's production LLM training cluster, GPUs with hardware defects silently corrupted computation without any error signal. Training jobs either crashed with errors that looked like software bugs, such as a shape mismatch or an out-of-bound access, or kept running with sudden loss spikes until engineers stopped them by hand. Of the 23 SDC-defective GPUs the paper characterises, 13 surfaced this way during active LLM training and 10 were found in offline stress tests that used online workloads and failed numerical validation checks.

   Training divergence or loss spikeTraining job failure

   Root cause

   Hardware failure

   Component

   GPU hardware and memory

   Contributing factor

   Monitoring blind spot

   Detection

   Corruptions surfaced as crashes, NaN or loss spikes in training jobs, or as failed numerical validation in offline stress tests that used online workloads; ECC, thermal and other hardware signals did not show them.

   Remediation

   • Halted the training and took the affected nodes offline to confirm the defective GPU
   • Isolated the suspicious data-parallel group and resumed the training job from a checkpoint on healthy machines
   • Confirmed the defective GPU offline by replaying the alert-triggering trace and checking it with hardware tools
   • Sent the confirmed defective GPUs to hardware repair
   • Enabled every deterministic setting in production training, including torch.use_deterministic_algorithms, so anomalies could be replayed bit for bit

   Lessons

   SDC diagnosis has to be part of the training control plane rather than a separate hardware test: the most useful step during an incident is to take suspicious machines out of the training path quickly and leave device-level confirmation to offline analysis. Replaying the exact workload works better than synthetic stress tests, because many defective GPUs fail only on the kernels, data ranges and communication schedules that triggered the original anomaly, and because occurrence rates vary so widely a fixed-duration stress test is not a reliable acceptance check; production systems should keep the alert-triggering trace. Deterministic training turns ambiguous symptoms such as loss spikes into reproducible comparisons, and SDCs appear throughout the hardware lifecycle, so GPUs need continuous monitoring rather than checks only when new.

   View case
2. IL-0478 A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training steps

   Amazon2026

   Symptoms

   In Amazon's large-scale foundation model training, training steps took about 0.3 seconds longer (8.7 s instead of 8.4 s) when a node had a GPU whose network adapter had failed. No hardware failure alert fired, and standard node checks such as GPU burn-in and short NCCL tests rarely catch such nodes.

   Throughput degradation

   Trigger

   The network adapter of one GPU on a node failed.

   Root cause

   Hardware failure

   Component

   NCCLNIC, cable and switch port link

   Contributing factor

   Monitoring blind spot

   Detection

   No hardware alert or standard health check showed the fault. It showed as slower training steps, in the per-node network device status and transmission rate that Guard monitors, and as elevated latency or reduced bandwidth in a 2-node sweep.

   Remediation

   • Resolved the degraded communication path, which brought training step time from 8.7 s back to 8.4 s

   Action items

   • Monitor each node's network device status continuously to find and isolate nodes whose traffic NCCL rerouted
   • Remove a node flagged by online monitoring from the healthy node pool and schedule it for offline verification
   • Run a multi-node sweep that stresses cross-node collective communication to expose degraded links and misrouted traffic
   • Trigger node sweeps after anomalies found by online monitoring and after repairs, before a node re-enters production

   Lessons

   NCCL's transparent rerouting keeps both the job and NCCL tests working while it hides a failed adapter, so checks that verify functional correctness do not find the node. Amazon relies on performance-aware monitoring during training, with step time as the main signal, and on offline verification under realistic load before nodes return to service.

   View case
3. IL-0421 Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023

   Alibaba Group2025

   Symptoms

   Large LLM training jobs on Alibaba's AI clusters crashed again and again and had to restart from the last valid checkpoint. One of the largest jobs, on 4,096 GPUs in a newly deployed cluster, crashed 40 times in one month, and most crashes reached users only as "NCCL Errors".

   Training job failureNCCL timeoutGPU unavailable

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLPyTorch

   Contributing factor

   Monitoring blind spotUnverified assumption

   Detection

   Before C4D, users found crashes through NCCL errors and could stay unaware of a stalled job for up to 30 minutes until the PyTorch elastic agent killed it; C4D later detects hangs and slowdowns from collective communication timings within tens of seconds.

   Remediation

   • Allocated 64 backup GPUs on 8 servers for every 1024 GPUs on 128 servers, so that parallel training on any 128 servers of the 136-server pool keeps the same communication and performance
   • Deployed C4D to detect faulty nodes automatically, isolate them and restart the job from the most recent valid checkpoint
   • Saved checkpoints more often, every 10 minutes, to cut the work lost after each crash
   • Enhanced cooling with higher fan speeds and better air conditioning to keep GPUs from overheating
   • Identified and enhanced the most vulnerable components of the system, lowering the average error rate 3.33 times

   Action items

   • Take load variation into account in C4D for more precise slow detection

   Lessons

   Most crash causes are confined to specific nodes or devices, so isolating the faulty component quickly and restarting, while leaving root cause analysis for offline work, removes most of the error-induced downtime. Because synchronous training makes many different faults look like the same communication error, diagnosis has to use the regular rhythm of the workers at synchronisation points rather than the error message, and frequent checkpoints matter because error rates in large clusters are high.

   View case
4. IL-0479 Accelerator nodes with silent data corruption changed LLM training outputs, gradients and weights with no error signal

   Amazon Web Services2025

   Symptoms

   Machine learning accelerator nodes that a cloud computing platform's automated fleet management had removed from production produced wrong computation results without any error signal. When the same LLM training ran on one of them and on a healthy node, the training loss was identical, but submodule outputs, gradients and model weights differed.

   Training divergence or loss spike

   Root cause

   Hardware failure

   Component

   –

   Contributing factor

   Monitoring blind spot

   Detection

   The fleet management tests found the nodes: collective communication and compute unit stress tests and a small LLM training run compared with golden values, run after installation and when a node returns from customer use. Training loss showed nothing on most nodes.

   Remediation

   • Kept the nodes that failed the fleet's SDC tests out of production availability
   • Sent the nodes held for the study for repair before returning them to the production pool

   Lessons

   SDC can change training for an unknown period before any loss spike appears, so watching the training loss is not enough. ABFT did not detect it reliably; the authors' possible explanations are that mismatches were too rare and small to exceed floating-point error bounds, that ABFT lowered accelerator utilization and with it the SDC rate, and that SDC may arise outside matrix multiplication; the authors suggest recomputation, for example a shadow data-parallel replica whose gradients are compared with a target rank's during the gradient all-reduce, and point at reducing the sharpness of the loss surface to limit the damage.

   View case
5. IL-0411 Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checks

   Meta2024

   Symptoms

   In Meta's RSC-1 and RSC-2 research GPU clusters, certain nodes had above-average rates of job failures. After each failure the same nodes kept attracting new jobs and failing them too, while health checks and repair workflows did not flag them. Meta calls these servers lemon nodes.

   Training job failure

   Root cause

   Hardware failure

   Component

   GPU hardware and memorySlurm job scheduler

   Contributing factor

   Monitoring blind spot

   Detection

   Meta's lemon detection flags nodes from their history in per-node signals such as excl_jobid_count, xid_cnt, tickets, out_count and node failure counts, because the periodic health checks and repair workflows did not catch them.

   Remediation

   • Built lemon detection to proactively identify, isolate and replace lemon nodes in the Slurm scheduler
   • Identified 40 faulty nodes, 24 on RSC-1 and 16 on RSC-2

   Lessons

   Finding defective nodes needs historic data, and lemon node detection can improve the completion rate of large jobs by over 30%. A defective node that cannot be removed from capacity leads to restart loops that make large gang-scheduled jobs impossible to run, so recovery from random failures only works once defective nodes are reliably rotated out.

   View case

By root cause1