Cases4 cases

1 to 4 of 4

1. IL-0416 Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errors

   ByteDance2026

   Symptoms

   In ByteDance's production LLM training cluster, GPUs with hardware defects silently corrupted computation without any error signal. Training jobs either crashed with errors that looked like software bugs, such as a shape mismatch or an out-of-bound access, or kept running with sudden loss spikes until engineers stopped them by hand. Of the 23 SDC-defective GPUs the paper characterises, 13 surfaced this way during active LLM training and 10 were found in offline stress tests that used online workloads and failed numerical validation checks.

   Training divergence or loss spikeTraining job failure

   Root cause

   Hardware failure

   Component

   GPU hardware and memory

   Contributing factor

   Monitoring blind spot

   Detection

   Corruptions surfaced as crashes, NaN or loss spikes in training jobs, or as failed numerical validation in offline stress tests that used online workloads; ECC, thermal and other hardware signals did not show them.

   Remediation

   • Halted the training and took the affected nodes offline to confirm the defective GPU
   • Isolated the suspicious data-parallel group and resumed the training job from a checkpoint on healthy machines
   • Confirmed the defective GPU offline by replaying the alert-triggering trace and checking it with hardware tools
   • Sent the confirmed defective GPUs to hardware repair
   • Enabled every deterministic setting in production training, including torch.use_deterministic_algorithms, so anomalies could be replayed bit for bit

   Lessons

   SDC diagnosis has to be part of the training control plane rather than a separate hardware test: the most useful step during an incident is to take suspicious machines out of the training path quickly and leave device-level confirmation to offline analysis. Replaying the exact workload works better than synthetic stress tests, because many defective GPUs fail only on the kernels, data ranges and communication schedules that triggered the original anomaly, and because occurrence rates vary so widely a fixed-duration stress test is not a reliable acceptance check; production systems should keep the alert-triggering trace. Deterministic training turns ambiguous symptoms such as loss spikes into reproducible comparisons, and SDCs appear throughout the hardware lifecycle, so GPUs need continuous monitoring rather than checks only when new.

   View case
2. IL-0479 Accelerator nodes with silent data corruption changed LLM training outputs, gradients and weights with no error signal

   Amazon Web Services2025

   Symptoms

   Machine learning accelerator nodes that a cloud computing platform's automated fleet management had removed from production produced wrong computation results without any error signal. When the same LLM training ran on one of them and on a healthy node, the training loss was identical, but submodule outputs, gradients and model weights differed.

   Training divergence or loss spike

   Root cause

   Hardware failure

   Component

   –

   Contributing factor

   Monitoring blind spot

   Detection

   The fleet management tests found the nodes: collective communication and compute unit stress tests and a small LLM training run compared with golden values, run after installation and when a node returns from customer use. Training loss showed nothing on most nodes.

   Remediation

   • Kept the nodes that failed the fleet's SDC tests out of production availability
   • Sent the nodes held for the study for repair before returning them to the production pool

   Lessons

   SDC can change training for an unknown period before any loss spike appears, so watching the training loss is not enough. ABFT did not detect it reliably; the authors' possible explanations are that mismatches were too rare and small to exceed floating-point error bounds, that ABFT lowered accelerator utilization and with it the SDC rate, and that SDC may arise outside matrix multiplication; the authors suggest recomputation, for example a shadow data-parallel replica whose gradients are compared with a target rank's during the gradient all-reduce, and point at reducing the sharpness of the loss surface to limit the damage.

   View case
3. IL-0238 Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 cluster

   DeepSeek2024

   Symptoms

   On DeepSeek's Fire-Flyer 2 training cluster of 10,000 PCIe A100 GPUs, Xid 74 NVLink errors, which occur mainly on the NVLink Bridge between pairs of GPUs, were the most common GPU Xid error and occurred several orders of magnitude more often than other hardware faults. InfiniBand link flash cuts disrupted application communication and sometimes failed training tasks, and computational and GPU memory errors that ECC did not detect caused gradnorm spikes, loss explosions and even non-convergence of models.

   GPU unavailableLink flappingTraining job failureTraining divergence or loss spike

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryInfiniBand fabric

   Contributing factor

   –

   Detection

   Xid errors were read from the NVIDIA driver's messages in the kernel or event log. A validator run weekly by the automatic operations system tested every node, and diagnosis tools such as hostping were integrated into the platform.

   Remediation

   • Reset the GPU to clear most GPU memory ECC errors, relying on A100 row remapping
   • Reset the GPU or rebooted the node after uncorrectable GPU errors
   • Ran a fieldiag test after Xid 119 and sent most such GPUs for RMA
   • Removed nodes that failed the weekly validator from the scheduling platform
   • Resumed interrupted training from checkpoints saved about every 5 minutes

   Lessons

   The best way to keep devices stable is to find problems before they happen, which is why the validator runs weekly. GPU ECC faults far outnumber CPU ones and need to be handled quickly so that performance and accuracy are not affected. Software-looking Xid errors such as illegal memory access may come from corrupted memory, so hardware faults should be considered once software bugs are ruled out. Whether to install NVLink should depend on actual need because of its drawbacks.

   View case
4. IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errors

   Shanghai AI Laboratory2024

   Symptoms

   Over six months from March to August 2023, LLM pretraining jobs on the Seren and Kalos clusters of Shanghai AI Laboratory's Acme datacenter, 4,704 A100 GPUs in total, failed again and again partway through and had to be restarted from the last checkpoint. Jobs that failed on infrastructure problems were only 11% of failed jobs by count but used over 82% of the GPU time of failed jobs.

   Training job failureGPU unavailableNCCL timeoutTraining divergence or loss spike

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLTraining data loaderObject storageData center power and cooling

   Contributing factor

   –

   Detection

   Failures were found in the framework's runtime logs, later by a diagnosis system that combines rules with an LLM; the pretraining framework detects loss spikes automatically. Jobs that stalled without an error were noticed only when users inspected them by hand.

   Remediation

   • Restarted interrupted pretraining jobs by hand from the last checkpoint, with developers on call in turn
   • Saved checkpoints at shorter intervals and let jobs terminate gracefully to keep their current state
   • Improved the cluster's cooling after GPU overheating caused NVLink and ECC errors
   • Found faulty nodes with a two-round NCCL allgather test and cordoned them off
   • Restarted from an earlier healthy checkpoint and skipped the following data batches after a loss spike
   • Fixed the garbage collection interval on each rank in InternEvo V2
   • Set num_worker=0 and enabled garbage collection in the dataloader to stop the memory leak

   Action items

   • Move failure diagnosis from GPT-4 to Shanghai AI Laboratory's own LLMs
   • Upgrade the infrastructure, with a particular focus on the NICs, and expand the computing cluster for larger-scale pretraining

   Lessons

   Infrastructure failures are rare by count but cost the most GPU time, so diagnosing them and recovering quickly matters most for training efficiency. Matching error logs with regular-expression rules alone is often inaccurate because the logs are diverse and one failure can print several errors. Jobs that stall without an error or that are paused for loss spikes waste resources until someone looks, which calls for a failure-handling system that finds and fixes faulty jobs by itself.

   View case

By root cause1