IL-0025 Imbue · 2024-06-25 (published)ai-infranetworking
Imbue's write-up is a guide to the infrastructure it built to train a 70B-parameter model, not a report on one outage. It covers one cluster of 4,088 H100 GPUs on 511 hosts, where each GPU has its own ConnectX-7 card at 400 Gbps on a three-tier, fully non-blocking InfiniBand network, set up with Voltage Park, Dell, H5 and NVIDIA. Training at full speed needed every one of over 12,000 connections to work.
Machines were provisioned through Dell iDRAC and Ubuntu MAAS. The MAAS integration with the BMC was unreliable, so MAC addresses were collected through the iDRAC API in advance, and on the first provisions the clocks were so far off that HTTPS certificate validation stopped apt from installing anything. About 10% of hosts failed to boot for hardware reasons and were sent back to Dell or to data center staff. GPU errors were mostly fixed by reseating the cards, the internal PCIe cables had to be reseated on about a quarter of the hosts, Dell fixed a number of rarer faults with firmware upgrades, and a bad NVIDIA driver behind wrong temperature readings was downgraded to the previous version.
The top tier of the InfiniBand fabric had been designed as eight disconnected networks with no common routing paths. It was rewired, and checks were added that the physical connections matched the new design. Hot air recirculating through open spaces between switches in the same racks made ports run hot until the data center partners put in a workaround. Flapping and erroring ports were cleaned and reseated, and the remaining alerting transceivers were disabled while waiting for replacements. A brute-force burn-in that pushed as much data as possible through every port at once (over 97% of theoretical capacity on most ports, with some switches crashing) separated the ports robust enough to keep from those to disable or repair. GPUDirect RDMA needed an extra kernel module and PCIe Access Control Service (ACS) disabled to avoid immediate hangs. Later, large regressions came from firmware upgrades, especially ones applied to only half the cluster, and from mass restarts of GPU hosts, both of which could corrupt or flood the UFM state and require UFM restarts.
For training, Imbue wrote host health checks that answer yes or no for each machine, forked NCCL to log which operations were in flight when a run stalled, live-debugged stalled processes with Py-Spy and GDB, and relaunched crashed runs automatically from the last checkpoint on the healthiest hosts. Live debugging caught one hang caused by a misconfiguration in the Python threading settings that made the eight multi-threaded NCCL GPU processes hit a race condition during pre-PyTorch initialization on certain hosts. The gradual MFU sag came from automatic garbage collection: in synchronous FSDP training, one worker running garbage collection during a blocking operation slows every other worker, and disabling automatic collection in favour of collection at set intervals on all hosts removed the sags. Network failures are handled by parsing the UFM event log, disabling the implicated links and ports, filing maintenance tickets and re-enabling components after repair.
When Imbue brought up a cluster of 4,088 H100 GPUs on 511 hosts, about 10% of the machines failed to boot, mostly from physical problems: unconnected or miswired Ethernet cables, iDRAC hardware issues, broken power supplies, bad NVMe drives, missing internal wires, and network cards or GPUs that did not show up. In single-node GPU tests, Ubuntu server logs showed links between the GPUs and the PCIe buses or network cards reporting "limited width: x4 < x16". A few hosts had NVMe drives that locked up the whole machine when touched, drives that appeared in random order and got the OS installed on the wrong one, wrong temperature readings that kept fans at 100%, and CPUs limited to 2 GHz.
On the InfiniBand fabric the Unified Fabric Manager (UFM) at first could not detect any of the 320 switches. Once it could, almost every switch port reported temperatures sometimes above 70 degrees Celsius before carrying any data. Under real traffic many ports showed high error rates or flapped between working and broken, about eighteen hundred alerts, and these only appeared when the ports were in use. Once about 10% of the fabric had issues, adaptive routing could no longer route reliably around the dropping links.
Training runs crashed on startup with errors such as "Forward order differs across ranks: rank 0 is all-gathering 43 parameters while rank 1228 is all-gathering 1 parameters" and "CUDA out of memory. Tried to allocate …", and mid-run on Xid and SXid hardware faults that left no meaningful Python stack trace. The hardest were hangs, where every GPU worker printed at the same time a message like "Watchdog caught collective operation timeout: WorkNCCL(SeqNum=408951, OpType=_ALLGATHER_BASE, … , Timeout(ms)=600000) ran for 600351 milliseconds before timing out". One or more hosts had failed to finish an NCCL operation and all the others blocked until NCCL_TIMEOUT, and the nature of NCCL made it very hard to tell which host was at fault; finding it often meant looking for the host that did not produce log messages the others did.
Slowdowns showed as model FLOPs utilization (MFU) below what had been reached before, and the shape of the per-batch MFU curve pointed to the kind of fault. Under a tenth of the expected MFU from the start was most often an InfiniBand hardware fault such as a dead T2 or T3 switch, or a fault between GPU and NIC; 30% could be one host with GDR set up wrongly; 60 to 80% was most commonly degraded or faulty InfiniBand links, and could also be CPU throttling. Dips to 70% every 15 seconds matched NVIDIA clock throttle reasons from heat or power supply faults, and a gradual sag over a run that returned to 100% on restart was traced to automatic garbage collection.
Symptom
Root cause Dependency failure
Automated hardware checks during provisioning, UFM alerts on temperature, error rates and flapping ports, Prometheus node exporter and DCGM metrics, per-batch MFU monitoring, and live debugging of stalled processes with Py-Spy, GDB and NVIDIA Nsight.
Keep 10 to 20% more machines than a run needs, and network them so any working subset can be used. On the newest hardware expect about 3% of machines to break each week, mostly the same few machines breaking in different ways, so growing a set of known reliable "golden" hosts works better than chasing random failures. Write tests and automation for every kind of failure met, because each one comes back, and write tools that make opaque errors readable. Change only one thing at a time, and double-check the claims of new tools and people before building on them.
IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026same root cause · shared InfiniBand fabric, NCCL, PyTorch
IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021same root cause · shared InfiniBand fabric, NCCL
IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026same root cause · shared InfiniBand fabric
IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMsLambda · 2026 · sev1same root cause · shared InfiniBand fabric
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared GPU driver