IL-0480 Samsung SDS · 2026-08-06 (published)ai-infra

Epoch-end NCCL deadlock hung a 16-GPU B300 fine-tuning run for three hours while GPU utilization read 100%

Samsung SDS engineers fully fine-tuned Qwen3-32B on two nodes of 8 NVIDIA B300 GPUs with PyTorch FSDP in FULL_SHARD mode (equivalent to ZeRO-3), InfiniBand NDR between the nodes and an NGC container with PyTorch 2.12, writing checkpoints to NFS. The data loader sharded records evenly across ranks, tokenized each record once and packed the tokens into fixed 2048-token blocks in a node-local file, and each rank trained on its own blocks. Because the records differ in length, the block counts differed after packing: rank 3 had the fewest (30,835) and rank 11 the most (30,888), a spread of 53 blocks or 0.17% of the per-rank work.

In the run that failed, each rank ran until its own blocks were used up. Rank 3 finished about 7 optimizer steps before rank 11, left the loop and entered the checkpoint barrier while the others were still reducing gradients, and the job deadlocked. The paper notes that this is documented PyTorch behaviour behind the Join context manager, but Join could not be used: neither FSDP1 nor FSDP2 implements Joinable, and the deadlock was at a manual dist.barrier() outside Join's scope. The team added evenfix, which all-reduces the block counts to the global minimum before the loop so that every rank stops at the same step, dropping at most 53 blocks on any rank, about 0.085% of all tokens. The failure cost about 48 GPU·h for the hang and about 85 GPU·h of progress that could not be recovered, because that run saved only at epoch end, about 130 GPU·h in total. A clean epoch afterwards ran 3,854 steps and its epoch-end save completed with no watchdog, abort or deadlock traces.

Severity: –
Scope: 16-GPU B300 fine-tuning run hung for about 3 hours and lost about 5 hours of progress, about 130 GPU·h
Users affected: –
Region: –
Time to detect: 3h
Time to resolve: –

Symptoms

A Qwen3-32B full fine-tuning run on 16 NVIDIA B300 GPUs in two nodes trained normally for about 5 hours, then hung at step 3,850 of 3,857 as it entered the end-of-epoch checkpoint. There was no error, no OOM in dmesg and no segfault, nvidia-smi still showed 100% GPU utilization although throughput was exactly zero, and the hang lasted three hours until the NCCL watchdog surfaced it.

Before the hang the run looked healthy: loss went from 1.60 to 0.74 to about 0.68, throughput was about 53k tok/s, and InfiniBand bandwidth was nominal. During the hang board power fell to about 190 W per GPU, with one rank at 0% utilization and the other fifteen at 100%. A normal checkpoint also sits near 190 W, but then only rank 0 drops to 0% while it writes the checkpoint, and the run resumes within about 2 minutes.

The NCCL flight recorder, enabled with TORCH_NCCL_TRACE_BUFFER_SIZE, showed rank 3 inside an all_reduce, the dist.barrier() that guards the end-of-epoch save_checkpoint, while ranks 0–2 and 4–15 were blocked in a reduce_scatter of the backward pass. With two different collectives at the same slot, NCCL waited forever.

Smoke tests had passed. A run limited with max_steps=N exercises only the training loop, where all ranks behave the same, and ends before the rank with the least data runs out, so the fault could appear only at the end of a full epoch.

Symptom NCCL timeout Training job failure

Root cause

Root cause Software defect

Component NCCL PyTorch Training data loader

Contributing factor Gap in automated tests Monitoring blind spot

Trigger: Rank 3, which had 53 fewer packed blocks than rank 11, ran out of data at the end of the epoch and entered the checkpoint barrier while the other ranks were still in the backward pass.

Detection

The NCCL watchdog surfaced the hang after three hours. The team then classified it as a deadlock from nvidia-smi board power and per-rank utilization and confirmed it with the NCCL flight recorder.

Remediation

Added evenfix, which all-reduces each rank's packed-block count to the global minimum before the loop so that every rank stops at the same step
Ran a clean epoch with evenfix in which the epoch-end save completed without a deadlock

Action items

Run a preflight gate in the launcher that checks six invariants, including per-rank block balance, and refuses to start an unsafe run
Watch each run from a separate host for completion, crash, hang, the power deadlock signature and a throughput floor, and send an out-of-band alert
Save checkpoints periodically, every 500 steps, instead of only at epoch end
Resume from weights only
Raise the watchdog timeout from 10 to 30 minutes
Retry failed launches automatically in the launcher

Lessons

GPU utilization reads 100% during an NCCL hang, while board power and the per-rank pattern show what the GPUs are really doing. For data-parallel jobs whose per-rank work depends on the data, a passing smoke test does not show that the full run is safe; the team now verifies invariants such as per-rank block balance before launch and watches runs from outside.

Sources

https://arxiv.org/abs/2608.05944 · paper · archived copy

References

Related cases

IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobsInfrawaves · 2026
same root cause · shared NCCL

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, NCCL, PyTorch, Training data loader among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0480
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0480/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.