Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

InfiniBand fabric

InfiniBand 패브릭 · infiniband

5 cases · 2021 to 2026

Root cause Dependency failure5

Contributing factor Monitoring blind spot3 No runbook2 Unverified assumption1 Unreviewed defaults1 Limited debug access1 No alert on limit1 No headroom1 Overlapping failures1 Partial rollout1 Single point of failure1

Symptom GPU unavailable3 Training job failure3 Link flapping2 NCCL timeout2

Organisation Databricks1 Imbue1 Lambda1 Meta AI1 Nebius1

Cases, by severity

IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMsLambda · 2026 · sev1
Root cause Dependency failure
IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026
Root cause Dependency failure
A training run crashed with an NCCL timeout after one InfiniBand port went down once and recovered

View on the graph

Control plane unresponsive1

Monitoring outage1

Service outage1

Storage disconnected or stale mounts1

Throughput degradation1

IL-0023

Databricks · 2026

Root cause Dependency failure

IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024

Root cause Dependency failure

IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021

Root cause Dependency failure