IL-0023 Databricks · 2026-07-01 (published)ai-infra
Databricks describes this incident in a post on how it keeps GPUs reliable across its AI platform. A training run failed with an NCCL timeout seven hours in, and the cause was one InfiniBand port used for RDMA NCCL collectives that had gone down once and come back.
The crash came down to which of two NCCL timeouts fires first. The PyTorch NCCL watchdog timeout, set with init_process_group(timeout=...), kills a hung collective after a configurable time, typically 10 minutes. Lower in the stack, NCCL_IB_TIMEOUT at the InfiniBand transport layer controls how long a connection waits for a downed port to recover before tearing the connection down; with retries its effective default is roughly seven seconds, much shorter than most teams realise. Once the port-down window exceeded that, the connection was gone and the collective was already dead regardless of the watchdog setting, and by the time the watchdog noticed the hang the run was bound to crash.
Databricks tuned its NCCL_IB_TIMEOUT defaults to be more resilient, and uses the same port-down signal to crash the job and restart it from a checkpoint instead of leaving GPUs idle until the watchdog fires. Its gpu-monitor service, which runs on every GPU node, thresholds RDMA fabric port-down detection on cumulative downtime rather than on the number of flaps, because one long enough flap can crash a multi-day run just as repeated flaps over hours can.
Seven hours into a distributed training run on Databricks AI, the job failed with an NCCL timeout.
In distributed training almost every underlying fault, whether a GPU degrading or falling off the bus, an RDMA fabric problem, an I/O hang or a CPU-side rank diverging from the others, looks the same from the job's side: every rank blocks on the same collective until the NCCL watchdog kills the job, and the logs show the NCCL watchdog timeout message. That message says almost nothing about the root cause, and the stack trace only shows the symptom.
The investigation found that a single InfiniBand port used for RDMA NCCL collectives had gone down once and recovered, and never flapped again. The continuous health checks did monitor InfiniBand port flapping, but a single isolated flap does not normally indicate an unhealthy port, so it did not trip the threshold and nothing had flagged it.
Root cause Dependency failure
The training job crashed with the NCCL timeout. The continuous health checks, which watched InfiniBand port flapping by count, had not flagged the single flap.
For training impact the signal that matters is cumulative port downtime, not the number of flaps: one long enough flap can crash a multi-day run. NCCL_IB_TIMEOUT, with an effective default of about seven seconds, decides the outcome long before the PyTorch watchdog timeout does. Health checks need to cover the whole node lifecycle: active checks before workloads start, passive checks for silent degradation under load, and periodic multi-node NCCL fabric checks in between.
IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024same root cause · shared InfiniBand fabric, NCCL, PyTorch
IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021same root cause · shared InfiniBand fabric, NCCL
IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026same root cause · shared InfiniBand fabric
IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMsLambda · 2026 · sev1same root cause · shared InfiniBand fabric
IL-0025 linked by the editorhttps://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.