Cases1 case

1 to 1 of 1

1. IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

   Imbue2024

   Symptoms

   When Imbue brought up a cluster of 4,088 H100 GPUs on 511 hosts, about 10% of the machines failed to boot, mostly from physical problems: unconnected or miswired Ethernet cables, iDRAC hardware issues, broken power supplies, bad NVMe drives, missing internal wires, and network cards or GPUs that did not show up. In single-node GPU tests, Ubuntu server logs showed links between the GPUs and the PCIe buses or network cards reporting "limited width: x4 < x16". A few hosts had NVMe drives that locked up the whole machine when touched, drives that appeared in random order and got the OS installed on the wrong one, wrong temperature readings that kept fans at 100%, and CPUs limited to 2 GHz.

   Link flappingNCCL timeoutThroughput degradationGPU unavailableTraining job failure

   Trigger

   Bringing a new InfiniBand-connected H100 cluster into service and running the first multi-node training jobs on it.

   Root cause

   Dependency failure

   Component

   InfiniBand fabricNCCLGPU driverPyTorch

   Contributing factor

   Unverified assumptionMonitoring blind spotPartial rollout

   Detection

   Automated hardware checks during provisioning, UFM alerts on temperature, error rates and flapping ports, Prometheus node exporter and DCGM metrics, per-batch MFU monitoring, and live debugging of stalled processes with Py-Spy, GDB and NVIDIA Nsight.

   Remediation

   • Automated checks for broken machines and sent faulty ones back to Dell or to data center staff
   • Reseated GPUs and internal PCIe cables and applied firmware upgrades
   • Downgraded the NVIDIA driver that caused wrong temperature readings
   • Rewired the top tier of the InfiniBand fabric into a single network and added checks that the wiring matched the design
   • Cleaned and reseated alerting ports and disabled the remaining alerting transceivers
   • Ran a burn-in workload through every InfiniBand port and disabled the ports that did not survive it
   • Enabled GPUDirect RDMA with an extra kernel module and PCIe ACS disabled
   • Disabled automatic garbage collection and scheduled it at set intervals on all hosts

   Action items

   • Release host-level health checks that confirm a host is free from known errors
   • Release an NCCL patch that improves logging around errors and stalls
   • Release a GPU stress test and NVLink and InfiniBand networking tests
   • Release a script that parses the UFM event log and decides which network ports to disable
   • Release a script that generates a burn-in workload exercising every InfiniBand link
   • Relaunch crashed runs automatically from the last checkpoint on the machines that pass health checks
   • Replace the disabled alerting transceivers

   Lessons

   Keep 10 to 20% more machines than a run needs, and network them so any working subset can be used. On the newest hardware expect about 3% of machines to break each week, mostly the same few machines breaking in different ways, so growing a set of known reliable "golden" hosts works better than chasing random failures. Write tests and automation for every kind of failure met, because each one comes back, and write tools that make opaque errors readable. Change only one thing at a time, and double-check the claims of new tools and people before building on them.

   View case

By root cause1