Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

PyTorch

PyTorch · pytorch

4 cases · 2022 to 2026

Root cause Dependency failure2 Hardware failure2

Contributing factor Unreviewed defaults2 Monitoring blind spot2 Unverified assumption1 Limited debug access1 Partial rollout1

Symptom Training job failure4 GPU unavailable3 NCCL timeout3 Link flapping2

Organisation BigScience1 Databricks1 Imbue1 Meta1

Cases, by severity

IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026
Root cause Dependency failure
IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024
Root cause Hardware failure
IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

View on the graph

Single point of failure1

Throughput degradation1

Imbue · 2024

Root cause Dependency failure

IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022

Root cause Hardware failure