Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Link flapping

링크 플래핑 · link-flapping

3 cases · 2024 to 2026

Root cause Dependency failure2 DNS or network fault1

Component NCCL3 InfiniBand fabric2 PyTorch2 GPU driver1 NIC, cable and switch port link1

Contributing factor Unreviewed defaults2 Monitoring blind spot2 Unverified assumption1 Partial rollout1

Organisation ByteDance1 Databricks1 Imbue1

Cases

IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026
Root cause Dependency failure
IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024
Root cause Dependency failure
IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUs

View on the graph

ByteDance · 2024

Root cause DNS or network fault