Cases2 cases

1 to 2 of 2

1. IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

   Imbue2024

   Symptoms

   When Imbue brought up a cluster of 4,088 H100 GPUs on 511 hosts, about 10% of the machines failed to boot, mostly from physical problems: unconnected or miswired Ethernet cables, iDRAC hardware issues, broken power supplies, bad NVMe drives, missing internal wires, and network cards or GPUs that did not show up. In single-node GPU tests, Ubuntu server logs showed links between the GPUs and the PCIe buses or network cards reporting "limited width: x4 < x16". A few hosts had NVMe drives that locked up the whole machine when touched, drives that appeared in random order and got the OS installed on the wrong one, wrong temperature readings that kept fans at 100%, and CPUs limited to 2 GHz.

   Link flappingNCCL timeoutThroughput degradationGPU unavailableTraining job failure

   Trigger

   Bringing a new InfiniBand-connected H100 cluster into service and running the first multi-node training jobs on it.

   Root cause

   Dependency failure

   Component

   InfiniBand fabricNCCLGPU driverPyTorch

   Contributing factor

   Unverified assumptionMonitoring blind spotPartial rollout

   Detection

   Automated hardware checks during provisioning, UFM alerts on temperature, error rates and flapping ports, Prometheus node exporter and DCGM metrics, per-batch MFU monitoring, and live debugging of stalled processes with Py-Spy, GDB and NVIDIA Nsight.

   Remediation

   • Automated checks for broken machines and sent faulty ones back to Dell or to data center staff
   • Reseated GPUs and internal PCIe cables and applied firmware upgrades
   • Downgraded the NVIDIA driver that caused wrong temperature readings
   • Rewired the top tier of the InfiniBand fabric into a single network and added checks that the wiring matched the design
   • Cleaned and reseated alerting ports and disabled the remaining alerting transceivers
   • Ran a burn-in workload through every InfiniBand port and disabled the ports that did not survive it
   • Enabled GPUDirect RDMA with an extra kernel module and PCIe ACS disabled
   • Disabled automatic garbage collection and scheduled it at set intervals on all hosts

   Action items

   • Release host-level health checks that confirm a host is free from known errors
   • Release an NCCL patch that improves logging around errors and stalls
   • Release a GPU stress test and NVLink and InfiniBand networking tests
   • Release a script that parses the UFM event log and decides which network ports to disable
   • Release a script that generates a burn-in workload exercising every InfiniBand link
   • Relaunch crashed runs automatically from the last checkpoint on the machines that pass health checks
   • Replace the disabled alerting transceivers

   Lessons

   Keep 10 to 20% more machines than a run needs, and network them so any working subset can be used. On the newest hardware expect about 3% of machines to break each week, mostly the same few machines breaking in different ways, so growing a set of known reliable "golden" hosts works better than chasing random failures. Write tests and automation for every kind of failure met, because each one comes back, and write tools that make opaque errors readable. Change only one thing at a time, and double-check the claims of new tools and people before building on them.

   View case
2. IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUs

   ByteDance2024

   Symptoms

   ByteDance's large language model training jobs on its MegaScale system, which runs on more than 10,000 GPUs, occasionally stalled or slowed down. The cause was network interfaces that flapped frequently.

   Link flappingNCCL timeoutThroughput degradation

   Trigger

   Network interfaces went down for several seconds and came back up, dropping every packet in transit.

   Root cause

   DNS or network fault

   Component

   NCCLNIC, cable and switch port link

   Contributing factor

   Unreviewed defaults

   Detection

   MegaScale's monitoring system uses second-level monitoring to track link flapping and other NIC issues. When the RDMA traffic metrics carried in the training daemons' heartbeats drop sharply or fluctuate abnormally, the driver raises an alert for manual investigation, and if the traffic stops entirely it starts the fault recovery procedure automatically.

   Remediation

   • set NCCL's timeout threshold explicitly to a larger value than the default
   • tuned NCCL's retransmit timer and retry count for fast recovery under link flapping
   • enabled the adap_retrans feature on the NICs
   • applied lower-level quality control to NIC signal strength, AOC cable quality and switch-side signal strength

   Lessons

   A default NCCL timeout can be shorter than the few seconds an interface needs to come back from a flap, so a brief link loss becomes a failed collective and a stalled job. A larger timeout and faster retransmission give the link time to return, but the flapping itself came from physical link quality between NIC, AOC cable and switch, and quality control at that level reduced it.

   View case

By root cause2