Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

NCCL timeout

NCCL 타임아웃 · nccl-timeout

4 cases · 2024 to 2026

Root cause Dependency failure2 DNS or network fault1 Hardware failure1

Component NCCL4 PyTorch3 InfiniBand fabric2 GPU driver1 GPU hardware and memory

Contributing factor Unreviewed defaults2 Monitoring blind spot2 Unverified assumption1 Partial rollout1

Organisation ByteDance1 Databricks1 Imbue1 Meta1

Cases

IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026
Root cause Dependency failure
IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024
Root cause Hardware failure
IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

View on the graph

1

NIC, cable and switch port link1

Single point of failure1

Imbue · 2024

Root cause Dependency failure

IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUsByteDance · 2024

Root cause DNS or network fault