Incident Library

Symptom

Training divergence or loss spike

학습 발산과 손실 급등 · training-divergence

4 cases · 2024 to 2026

Root cause Hardware failure4
Component GPU hardware and memory3Data center power and cooling1InfiniBand fabric1NCCL1Object storage1Training data loader1
Contributing factor Monitoring blind spot2
Organisation Amazon Web Services1ByteDance1DeepSeek1Shanghai AI Laboratory1

Cases

  1. IL-0416 Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026

    Root cause Hardware failure

  2. IL-0479 Accelerator nodes with silent data corruption changed LLM training outputs, gradients and weights with no error signalAmazon Web Services · 2025

    Root cause Hardware failure

  3. IL-0238 Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024

    Root cause Hardware failure

  4. IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024

    Root cause Hardware failure

View on the graph