Cases2 cases

1 to 2 of 2

1. IL-0416 Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errors

   ByteDance2026

   Symptoms

   In ByteDance's production LLM training cluster, GPUs with hardware defects silently corrupted computation without any error signal. Training jobs either crashed with errors that looked like software bugs, such as a shape mismatch or an out-of-bound access, or kept running with sudden loss spikes until engineers stopped them by hand. Of the 23 SDC-defective GPUs the paper characterises, 13 surfaced this way during active LLM training and 10 were found in offline stress tests that used online workloads and failed numerical validation checks.

   Training divergence or loss spikeTraining job failure

   Root cause

   Hardware failure

   Component

   GPU hardware and memory

   Contributing factor

   Monitoring blind spot

   Detection

   Corruptions surfaced as crashes, NaN or loss spikes in training jobs, or as failed numerical validation in offline stress tests that used online workloads; ECC, thermal and other hardware signals did not show them.

   Remediation

   • Halted the training and took the affected nodes offline to confirm the defective GPU
   • Isolated the suspicious data-parallel group and resumed the training job from a checkpoint on healthy machines
   • Confirmed the defective GPU offline by replaying the alert-triggering trace and checking it with hardware tools
   • Sent the confirmed defective GPUs to hardware repair
   • Enabled every deterministic setting in production training, including torch.use_deterministic_algorithms, so anomalies could be replayed bit for bit

   Lessons

   SDC diagnosis has to be part of the training control plane rather than a separate hardware test: the most useful step during an incident is to take suspicious machines out of the training path quickly and leave device-level confirmation to offline analysis. Replaying the exact workload works better than synthetic stress tests, because many defective GPUs fail only on the kernels, data ranges and communication schedules that triggered the original anomaly, and because occurrence rates vary so widely a fixed-duration stress test is not a reliable acceptance check; production systems should keep the alert-triggering trace. Deterministic training turns ambiguous symptoms such as loss spikes into reproducible comparisons, and SDCs appear throughout the hardware lifecycle, so GPUs need continuous monitoring rather than checks only when new.

   View case
2. IL-0476 PCIe downgrade on one machine slowed a 128-machine V100 training job for 40 minutes through a PFC surge

   ByteDance2025

   Symptoms

   A distributed training task on 128 machines in ByteDance's production environment slowed down severely for 40 minutes but kept running, holding thousands of V100 GPUs at reduced speed. The alerting in place notified engineers only when a task stopped completely, so the slowdown itself raised no alert.

   Throughput degradation

   Trigger

   The PCIe link of one machine degraded from 6.4Gbps to 4Gbps during the training task.

   Root cause

   Hardware failure

   Component

   PCIe linkInfiniBand fabric

   Contributing factor

   No alert on limit

   Detection

   No alert fired because alerts came only when a task stopped. Engineers from several teams found the faulty machine by manual diagnosis, which took 40 minutes.

   Action items

   • Detect faulty machines automatically during training by comparing each machine's per-second monitoring metrics with the other machines (Minder)

   Lessons

   Some faults slow a task without stopping it, and alerting that waits for a halted task misses them. Logs alone are incomplete because they do not contain monitoring metrics such as PFC packet rates, and no single metric reveals every fault type, so ByteDance compares machines with each other on several metrics and treats a difference that lasts for minutes as a fault.

   View case

By root cause1