Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Training job failure

학습 작업 실패 · training-job-failed

5 cases · 2021 to 2026

Root cause Dependency failure3 Hardware failure2

Component NCCL5 PyTorch4 InfiniBand fabric3 GPU hardware and memory2 GPU driver1

Contributing factor Unreviewed defaults2 Limited debug access2 Monitoring blind spot2 Unverified assumption1

Organisation BigScience1 Databricks1 Imbue1 Meta1 Meta AI1

Cases

IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026
Root cause Dependency failure
IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024
Root cause Hardware failure
IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

View on the graph

Object storage1

Slurm job scheduler1

Partial rollout1

Single point of failure1

Imbue · 2024

Root cause Dependency failure

IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022

Root cause Hardware failure

IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021

Root cause Dependency failure