Cases1 case

1 to 1 of 1

1. IL-0480 Epoch-end NCCL deadlock hung a 16-GPU B300 fine-tuning run for three hours while GPU utilization read 100%

   Samsung SDS2026

   Symptoms

   A Qwen3-32B full fine-tuning run on 16 NVIDIA B300 GPUs in two nodes trained normally for about 5 hours, then hung at step 3,850 of 3,857 as it entered the end-of-epoch checkpoint. There was no error, no OOM in dmesg and no segfault, nvidia-smi still showed 100% GPU utilization although throughput was exactly zero, and the hang lasted three hours until the NCCL watchdog surfaced it.

   NCCL timeoutTraining job failure

   Trigger

   Rank 3, which had 53 fewer packed blocks than rank 11, ran out of data at the end of the epoch and entered the checkpoint barrier while the other ranks were still in the backward pass.

   Root cause

   Software defect

   Component

   NCCLPyTorchTraining data loader

   Contributing factor

   Gap in automated testsMonitoring blind spot

   Detection

   The NCCL watchdog surfaced the hang after three hours. The team then classified it as a deadlock from nvidia-smi board power and per-rank utilization and confirmed it with the NCCL flight recorder.

   Remediation

   • Added evenfix, which all-reduces each rank's packed-block count to the global minimum before the loop so that every rank stops at the same step
   • Ran a clean epoch with evenfix in which the epoch-end save completed without a deadlock

   Action items

   • Run a preflight gate in the launcher that checks six invariants, including per-rank block balance, and refuses to start an unsafe run
   • Watch each run from a separate host for completion, crash, hang, the power deadlock signature and a throughput floor, and send an out-of-band alert
   • Save checkpoints periodically, every 500 steps, instead of only at epoch end
   • Resume from weights only
   • Raise the watchdog timeout from 10 to 30 minutes
   • Retry failed launches automatically in the launcher

   Lessons

   GPU utilization reads 100% during an NCCL hang, while board power and the per-rank pattern show what the GPUs are really doing. For data-parallel jobs whose per-rank work depends on the data, a passing smoke test does not show that the full run is safe; the team now verifies invariants such as per-rank block balance before launch and watches runs from outside.

   View case

By root cause1