IL-0416 ByteDance · 2026-07-13 (published)ai-infra
Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errors
The paper is written by ByteDance Seed and Shanghai Jiao Tong University engineers and characterises 23 GPUs with confirmed silent data corruption (SDC), errors that change data without any signal, collected from a large production LLM training cluster, by replaying the same workloads and kernels on the defective GPUs and measuring error rates. CPU and GPU memory are protected by ECC and were not the source. Only 25% of the corruptions were found during pre-deployment burn-in; 25% appeared within the first two months, 10% within six months and 40% around one year after deployment, and once a GPU started to corrupt data its error rate usually rose over time. Vendor analysis points to resistance drift in computational circuits caused by electrical stress from repeated intensive computation. The first corrupted kernel was GEMM in 17.3% of cases, InplaceMoE in 17.3%, FlashAttention in 13.0% and RMSNorm, InplaceLinear, Scatter, FusedKernel, Mean and internally implemented kernels in 8.69% each, and FP64 and FP32 CUDA cores were more prone to errors than Tensor Cores. The errors depended on data type and input value: two GPUs gave wrong FP32 GEMM results across all input ranges while another failed only in specific value bands, and one GPU gave 100% wrong FP64 results with 50 times the normal latency.
How often a defective GPU produced a wrong tensor when its alert-triggering workload was replayed ranged from 100% down to 10^-6 %, which is why generic stress tests missed so many; at a rate of 0.01%, diagnosing one GPU with the usual software replay would cost about as much compute as training three full LLM models (about 290%). Fail-stop errors left the allocated GPUs idle until the job was restored from a valid checkpoint and the defective hardware was isolated, while loss spikes could keep producing invalid checkpoints. ByteDance built SDCHunter, which runs production training deterministically (less than 0.01% step-time difference), splits the cluster along the data-parallel dimension into two replicas fed the same batch and compares hashes of the tensors at pipeline-parallel boundaries to find the divergent stage, then replays the iteration on that group and a healthy reference group to find the first divergent tensor and kernel and the GPU that produced it. SDCHunter is now the first hardware check when loss spikes or unexplained fail-stop errors occur; it has found 40 SDC-defective GPUs in production, the training job resumes within an hour after the suspicious group is isolated, and the defective GPU is confirmed offline in under an hour, where offline debugging used to take several days.
- Severity
- –
- Scope
- LLM training jobs crashed or produced corrupted checkpoints and had to be restored from a valid checkpoint; engineers spent days or weeks debugging software before the GPU was found
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
In ByteDance's production LLM training cluster, GPUs with hardware defects silently corrupted computation without any error signal. Training jobs either crashed with errors that looked like software bugs, such as a shape mismatch or an out-of-bound access, or kept running with sudden loss spikes until engineers stopped them by hand. Of the 23 SDC-defective GPUs the paper characterises, 13 surfaced this way during active LLM training and 10 were found in offline stress tests that used online workloads and failed numerical validation checks.
The first errors engineers saw were "Workload Test Error", "Suboptimal Convergence", "Unexpected CUDA Error", "Shape Error", "Value Error", "Program Crash" and "NaN in loss or output". In one Megatron MoE training job, a corruption during the token count aggregation silently changed the computed buffer size; the corrupted scatter operation then raised a shape mismatch, and in another case a corrupted modulo operation led to a segmentation fault on an out-of-bound access. The code at the crash site was correct, because the corruption had happened in an earlier kernel, so the error message pointed developers at the wrong place.
Nothing in the hardware monitoring showed the fault. Machine check exceptions, performance counters and device status registers on the defective GPUs looked the same as on healthy ones, the ECC error count had no correlation with the corruptions, and clock frequencies and thermal limits showed no anomaly. Some defects produced large deviations, while others were sparse bit flips; one GPU produced value biases smaller than 10^-3 that later operations could absorb.
Engineers first suspected the training software or numerical instability and could spend days or weeks isolating pipeline components; the hardware fault was usually found only after every software explanation had been ruled out. In one case, 10 training runs were started on the same cluster from the identical checkpoint: Runs 1 and 2 showed a sharp loss spike, Runs 3 and 4 trained normally, which made engineers suspect silent data corruption. Runs 5 and 6, and later Runs 8 and 9, spiked again at completely different steps, which pointed to a hardware fault in the floating-point units, so engineers halted the training and took the nodes offline. Synthetic stress tests and diagnostic tools such as intensive GEMM loops missed more than 60% of the defective GPUs.
Symptom Training divergence or loss spikeTraining job failure
Root cause
Root cause Hardware failure
- Trigger
- –
Detection
Corruptions surfaced as crashes, NaN or loss spikes in training jobs, or as failed numerical validation in offline stress tests that used online workloads; ECC, thermal and other hardware signals did not show them.
Remediation
- Halted the training and took the affected nodes offline to confirm the defective GPU
- Isolated the suspicious data-parallel group and resumed the training job from a checkpoint on healthy machines
- Confirmed the defective GPU offline by replaying the alert-triggering trace and checking it with hardware tools
- Sent the confirmed defective GPUs to hardware repair
- Enabled every deterministic setting in production training, including torch.use_deterministic_algorithms, so anomalies could be replayed bit for bit
Action items
–
Lessons
SDC diagnosis has to be part of the training control plane rather than a separate hardware test: the most useful step during an incident is to take suspicious machines out of the training path quickly and leave device-level confirmation to offline analysis. Replaying the exact workload works better than synthetic stress tests, because many defective GPUs fail only on the kernels, data ranges and communication schedules that triggered the original anomaly, and because occurrence rates vary so widely a fixed-duration stress test is not a reliable acceptance check; production systems should keep the alert-triggering trace. Deterministic training turns ambiguous symptoms such as loss spikes into reproducible comparisons, and SDCs appear throughout the hardware lifecycle, so GPUs need continuous monitoring rather than checks only when new.
Sources
Related cases
IL-0415GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025same root cause · shared GPU hardware and memory
IL-0482Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a dayNERSC · 2025same root cause · shared GPU hardware and memory
IL-0421Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025same root cause · shared GPU hardware and memory
IL-0414Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025same root cause · shared GPU hardware and memory
IL-0411Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checksMeta · 2024same root cause · shared GPU hardware and memory
IL-0238Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024same root cause · shared GPU hardware and memory
IL-0127Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024same root cause · shared GPU hardware and memory
IL-0239LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024same root cause · shared GPU hardware and memory
IL-0429GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022same root cause · shared GPU hardware and memory
IL-0427A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed tooBigScience · 2021same root cause · shared GPU hardware and memory
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0416
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0416/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.