Cases2 cases

1 to 2 of 2

1. IL-0480 Epoch-end NCCL deadlock hung a 16-GPU B300 fine-tuning run for three hours while GPU utilization read 100%

   Samsung SDS2026

   Symptoms

   A Qwen3-32B full fine-tuning run on 16 NVIDIA B300 GPUs in two nodes trained normally for about 5 hours, then hung at step 3,850 of 3,857 as it entered the end-of-epoch checkpoint. There was no error, no OOM in dmesg and no segfault, nvidia-smi still showed 100% GPU utilization although throughput was exactly zero, and the hang lasted three hours until the NCCL watchdog surfaced it.

   NCCL timeoutTraining job failure

   Trigger

   Rank 3, which had 53 fewer packed blocks than rank 11, ran out of data at the end of the epoch and entered the checkpoint barrier while the other ranks were still in the backward pass.

   Root cause

   Software defect

   Component

   NCCLPyTorchTraining data loader

   Contributing factor

   Gap in automated testsMonitoring blind spot

   Detection

   The NCCL watchdog surfaced the hang after three hours. The team then classified it as a deadlock from nvidia-smi board power and per-rank utilization and confirmed it with the NCCL flight recorder.

   Remediation

   • Added evenfix, which all-reduces each rank's packed-block count to the global minimum before the loop so that every rank stops at the same step
   • Ran a clean epoch with evenfix in which the epoch-end save completed without a deadlock

   Action items

   • Run a preflight gate in the launcher that checks six invariants, including per-rank block balance, and refuses to start an unsafe run
   • Watch each run from a separate host for completion, crash, hang, the power deadlock signature and a throughput floor, and send an out-of-band alert
   • Save checkpoints periodically, every 500 steps, instead of only at epoch end
   • Resume from weights only
   • Raise the watchdog timeout from 10 to 30 minutes
   • Retry failed launches automatically in the launcher

   Lessons

   GPU utilization reads 100% during an NCCL hang, while board power and the per-rank pattern show what the GPUs are really doing. For data-parallel jobs whose per-rank work depends on the data, a passing smoke test does not show that the full run is safe; the team now verifies invariants such as per-rank block balance before launch and watches runs from outside.

   View case
2. IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errors

   Shanghai AI Laboratory2024

   Symptoms

   Over six months from March to August 2023, LLM pretraining jobs on the Seren and Kalos clusters of Shanghai AI Laboratory's Acme datacenter, 4,704 A100 GPUs in total, failed again and again partway through and had to be restarted from the last checkpoint. Jobs that failed on infrastructure problems were only 11% of failed jobs by count but used over 82% of the GPU time of failed jobs.

   Training job failureGPU unavailableNCCL timeoutTraining divergence or loss spike

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLTraining data loaderObject storageData center power and cooling

   Contributing factor

   –

   Detection

   Failures were found in the framework's runtime logs, later by a diagnosis system that combines rules with an LLM; the pretraining framework detects loss spikes automatically. Jobs that stalled without an error were noticed only when users inspected them by hand.

   Remediation

   • Restarted interrupted pretraining jobs by hand from the last checkpoint, with developers on call in turn
   • Saved checkpoints at shorter intervals and let jobs terminate gracefully to keep their current state
   • Improved the cluster's cooling after GPU overheating caused NVLink and ECC errors
   • Found faulty nodes with a two-round NCCL allgather test and cordoned them off
   • Restarted from an earlier healthy checkpoint and skipped the following data batches after a loss spike
   • Fixed the garbage collection interval on each rank in InternEvo V2
   • Set num_worker=0 and enabled garbage collection in the dataloader to stop the memory leak

   Action items

   • Move failure diagnosis from GPT-4 to Shanghai AI Laboratory's own LLMs
   • Upgrade the infrastructure, with a particular focus on the NICs, and expand the computing cluster for larger-scale pretraining

   Lessons

   Infrastructure failures are rare by count but cost the most GPU time, so diagnosing them and recovering quickly matters most for training efficiency. Matching error logs with regular-expression rules alone is often inaccurate because the logs are diverse and one failure can print several errors. Jobs that stall without an error or that are paused for loss spikes waste resources until someone looks, which calls for a failure-handling system that finds and fixes faulty jobs by itself.

   View case

By root cause2