Cases2 cases

1 to 2 of 2

1. IL-0480 Epoch-end NCCL deadlock hung a 16-GPU B300 fine-tuning run for three hours while GPU utilization read 100%

   Samsung SDS2026

   Symptoms

   A Qwen3-32B full fine-tuning run on 16 NVIDIA B300 GPUs in two nodes trained normally for about 5 hours, then hung at step 3,850 of 3,857 as it entered the end-of-epoch checkpoint. There was no error, no OOM in dmesg and no segfault, nvidia-smi still showed 100% GPU utilization although throughput was exactly zero, and the hang lasted three hours until the NCCL watchdog surfaced it.

   NCCL timeoutTraining job failure

   Trigger

   Rank 3, which had 53 fewer packed blocks than rank 11, ran out of data at the end of the epoch and entered the checkpoint barrier while the other ranks were still in the backward pass.

   Root cause

   Software defect

   Component

   NCCLPyTorchTraining data loader

   Contributing factor

   Gap in automated testsMonitoring blind spot

   Detection

   The NCCL watchdog surfaced the hang after three hours. The team then classified it as a deadlock from nvidia-smi board power and per-rank utilization and confirmed it with the NCCL flight recorder.

   Remediation

   • Added evenfix, which all-reduces each rank's packed-block count to the global minimum before the loop so that every rank stops at the same step
   • Ran a clean epoch with evenfix in which the epoch-end save completed without a deadlock

   Action items

   • Run a preflight gate in the launcher that checks six invariants, including per-rank block balance, and refuses to start an unsafe run
   • Watch each run from a separate host for completion, crash, hang, the power deadlock signature and a throughput floor, and send an out-of-band alert
   • Save checkpoints periodically, every 500 steps, instead of only at epoch end
   • Resume from weights only
   • Raise the watchdog timeout from 10 to 30 minutes
   • Retry failed launches automatically in the launcher

   Lessons

   GPU utilization reads 100% during an NCCL hang, while board power and the per-rank pattern show what the GPUs are really doing. For data-parallel jobs whose per-rank work depends on the data, a passing smoke test does not show that the full run is safe; the team now verifies invariants such as per-rank block balance before launch and watches runs from outside.

   View case
2. IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobs

   Infrawaves2026

   Symptoms

   In Infrawaves' RoCEv2 GPU training clusters, LLM training jobs crashed because their communication collective collapsed after an RNIC port flapped. The Global Identifier (GID) of the RDMA network card's port, which in RoCEv2 is expected to stay the same after initialisation, had changed: with the default OS network manager, NetworkManager, the port briefly lost its IP address while it was down, and the GID number was incremented even though the original IP address came back. The paper describes the GID shift as non-deterministic.

   Link flappingTraining job failure

   Trigger

   Ports of the RDMA network cards (RNICs) on the GPU servers flapped and were briefly down.

   Root cause

   Software defect

   Component

   NIC, cable and switch port linkInfiniBand fabricNCCLHost routes and routing rules

   Contributing factor

   Unreviewed defaults

   Remediation

   • Configured the IP addresses with the Ubuntu Networking service instead of NetworkManager, so a port down no longer removed the address or changed the GID

   Lessons

   Host networking configuration is critical to the reliability of the collective communication library. In RoCEv2 a GID is expected to stay fixed after initialisation, but an IP configuration method that briefly drops the address when an RNIC port goes down can change it and bring down the training job. Changing the IP configuration method avoided the GID shift without the compatibility risks that driver-level patches carry in virtualised networks.

   View case

By root cause1