Incident Library

Organisation

Shanghai AI Laboratory

Shanghai AI Laboratory · shanghai-ai-laboratory

1 case · 2024

Root cause Hardware failure

Component Data center power and cooling GPU hardware and memory NCCL Object storage Training data loader

Contributing factor –

Symptom GPU unavailable NCCL timeout Training divergence or loss spike Training job failure

Cases

IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024
Root cause Hardware failure