Cases2 cases

1 to 2 of 2

1. IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recovered

   Databricks2026

   Symptoms

   Seven hours into a distributed training run on Databricks AI, the job failed with an NCCL timeout.

   Training job failureNCCL timeoutLink flapping

   Trigger

   A single InfiniBand port used for RDMA NCCL collectives went down once and recovered.

   Root cause

   Dependency failure

   Component

   NCCLInfiniBand fabricPyTorch

   Contributing factor

   Monitoring blind spotUnreviewed defaults

   Detection

   The training job crashed with the NCCL timeout. The continuous health checks, which watched InfiniBand port flapping by count, had not flagged the single flap.

   Remediation

   • Tuned the NCCL_IB_TIMEOUT defaults to be more resilient to short port-down windows
   • Used the port-down signal to crash the job and restart it from a checkpoint instead of waiting for the watchdog
   • Fed the investigation into gpu-monitor, whose RDMA fabric port-down check is thresholded on cumulative downtime rather than flap count

   Action items

   • Tune the NCCL_IB_TIMEOUT defaults to be more resilient
   • Crash and restart the job from a checkpoint on the port-down signal instead of leaving GPUs idle until the watchdog fires
   • Threshold RDMA fabric port-down detection on cumulative downtime, not flap count, in gpu-monitor

   Lessons

   For training impact the signal that matters is cumulative port downtime, not the number of flaps: one long enough flap can crash a multi-day run. NCCL_IB_TIMEOUT, with an effective default of about seven seconds, decides the outcome long before the PyTorch watchdog timeout does. Health checks need to cover the whole node lifecycle: active checks before workloads start, passive checks for silent degradation under load, and periodic multi-node NCCL fabric checks in between.

   View case
2. IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUs

   ByteDance2024

   Symptoms

   ByteDance's large language model training jobs on its MegaScale system, which runs on more than 10,000 GPUs, occasionally stalled or slowed down. The cause was network interfaces that flapped frequently.

   Link flappingNCCL timeoutThroughput degradation

   Trigger

   Network interfaces went down for several seconds and came back up, dropping every packet in transit.

   Root cause

   DNS or network fault

   Component

   NCCLNIC, cable and switch port link

   Contributing factor

   Unreviewed defaults

   Detection

   MegaScale's monitoring system uses second-level monitoring to track link flapping and other NIC issues. When the RDMA traffic metrics carried in the training daemons' heartbeats drop sharply or fluctuate abnormally, the driver raises an alert for manual investigation, and if the traffic stops entirely it starts the fault recovery procedure automatically.

   Remediation

   • set NCCL's timeout threshold explicitly to a larger value than the default
   • tuned NCCL's retransmit timer and retry count for fast recovery under link flapping
   • enabled the adap_retrans feature on the NICs
   • applied lower-level quality control to NIC signal strength, AOC cable quality and switch-side signal strength

   Lessons

   A default NCCL timeout can be shorter than the few seconds an interface needs to come back from a flap, so a brief link loss becomes a failed collective and a stalled job. A larger timeout and faster retransmission give the link time to return, but the flapping itself came from physical link quality between NIC, AOC cable and switch, and quality control at that level reduced it.

   View case

By root cause2