Cases13 cases

8 to 12 of 13

1. IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

   Imbue2024

   Symptoms

   When Imbue brought up a cluster of 4,088 H100 GPUs on 511 hosts, about 10% of the machines failed to boot, mostly from physical problems: unconnected or miswired Ethernet cables, iDRAC hardware issues, broken power supplies, bad NVMe drives, missing internal wires, and network cards or GPUs that did not show up. In single-node GPU tests, Ubuntu server logs showed links between the GPUs and the PCIe buses or network cards reporting "limited width: x4 < x16". A few hosts had NVMe drives that locked up the whole machine when touched, drives that appeared in random order and got the OS installed on the wrong one, wrong temperature readings that kept fans at 100%, and CPUs limited to 2 GHz.

   Link flappingNCCL timeoutThroughput degradationGPU unavailableTraining job failure

   Trigger

   Bringing a new InfiniBand-connected H100 cluster into service and running the first multi-node training jobs on it.

   Root cause

   Dependency failure

   Component

   InfiniBand fabricNCCLGPU driverPyTorch

   Contributing factor

   Unverified assumptionMonitoring blind spotPartial rollout

   Detection

   Automated hardware checks during provisioning, UFM alerts on temperature, error rates and flapping ports, Prometheus node exporter and DCGM metrics, per-batch MFU monitoring, and live debugging of stalled processes with Py-Spy, GDB and NVIDIA Nsight.

   Remediation

   • Automated checks for broken machines and sent faulty ones back to Dell or to data center staff
   • Reseated GPUs and internal PCIe cables and applied firmware upgrades
   • Downgraded the NVIDIA driver that caused wrong temperature readings
   • Rewired the top tier of the InfiniBand fabric into a single network and added checks that the wiring matched the design
   • Cleaned and reseated alerting ports and disabled the remaining alerting transceivers
   • Ran a burn-in workload through every InfiniBand port and disabled the ports that did not survive it
   • Enabled GPUDirect RDMA with an extra kernel module and PCIe ACS disabled
   • Disabled automatic garbage collection and scheduled it at set intervals on all hosts

   Action items

   • Release host-level health checks that confirm a host is free from known errors
   • Release an NCCL patch that improves logging around errors and stalls
   • Release a GPU stress test and NVLink and InfiniBand networking tests
   • Release a script that parses the UFM event log and decides which network ports to disable
   • Release a script that generates a burn-in workload exercising every InfiniBand link
   • Relaunch crashed runs automatically from the last checkpoint on the machines that pass health checks
   • Replace the disabled alerting transceivers

   Lessons

   Keep 10 to 20% more machines than a run needs, and network them so any working subset can be used. On the newest hardware expect about 3% of machines to break each week, mostly the same few machines breaking in different ways, so growing a set of known reliable "golden" hosts works better than chasing random failures. Write tests and automation for every kind of failure met, because each one comes back, and write tools that make opaque errors readable. Change only one thing at a time, and double-check the claims of new tools and people before building on them.

   View case
2. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs

   Google Cloud2024

   Symptoms

   From 08:00 US/Pacific on 12 March 2024, GKE GPU nodes running Container-Optimized OS (COS) that were newly created or recreated came up with GPU drivers that did not work, so workloads that used the GPUs could not run. GKE showed elevated errors.

   GPU unavailableNode NotReady

   Root cause

   Dependency failure

   Component

   GPU driverNode operating system imageNode pool

   Contributing factor

   Automation amplification

   Detection

   The report does not say how Google detected the issue. Affected users saw "Failed to download GPU driver installer, status: 403 Forbidden" on the GPU node when installing the driver.

   Remediation

   • Halted automatic node recreations, which attempt GPU driver downloads, until the issue was mitigated
   • Worked with the driver partner to resolve access to the driver download bucket

   Lessons

   The status page report states no lessons or follow-up actions.

   View case
3. IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1

   Crusoe Cloud2023

   Symptoms

   At 20:31 UTC on 15 November 2023, some GPU virtual machines in us-northcentral1-a became unavailable. One row of servers in a Crusoe Cloud data center in the us-northcentral1 region had lost power, and every server in the row powered off together with the VMs running on it.

   GPU unavailableService outage

   Trigger

   While the row's UPS was in bypass mode for routine maintenance, power switched from the primary source to a backup source for reasons unrelated to the maintenance.

   Root cause

   Dependency failure

   Component

   Data center power and cooling

   Contributing factor

   Single point of failureOverlapping failures

   Detection

   The power failure happened at 20:31 UTC. The first status post, at 21:07 UTC, said certain VMs in us-northcentral1-a were unavailable, and at 21:31 UTC the status page said the affected VMs had been identified.

   Remediation

   • identified the VMs on the row that lost power
   • restarted the affected VMs after power was restored

   Action items

   • Review the standard operating procedures for UPS maintenance to minimise customer impact during future maintenance
   • Build future data centers with redundant UPSs
   • Improve recovery procedures to reduce mean time to resolution

   Lessons

   When the only UPS for a row is in bypass for maintenance, the row is exposed to any power event during that window, even one unrelated to the work. Crusoe Cloud's response is redundant UPSs in future data centers, a review of maintenance procedures, and faster recovery of VMs after power returns.

   View case
4. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hung

   Meta AI2021

   Symptoms

   Between the team's 17 November 2021 update and 3 December 2021, the 175B-parameter OPT training run went down and had to be restarted more than 40 times, for hardware, infrastructure and training stability reasons. A hardware fault could stop the run for hours at any time of day, and several did over the Thanksgiving break.

   Training job failureGPU unavailable

   Trigger

   Nodes in the cloud training cluster failed with ECC errors, and each failure took the training run down.

   Root cause

   Dependency failure

   Component

   GPU hardware and memoryNCCLInfiniBand fabricObject storage

   Contributing factor

   No headroomLimited debug accessNo runbook

   Detection

   During these two weeks the team added monitoring of train.log, GPU burn-in testing to detect memory errors, and automated InfiniBand testing.

   Remediation

   • replaced failed nodes, and later did it through a script
   • added GPU burn-in testing to detect memory errors
   • automated InfiniBand testing
   • added monitoring of train.log
   • debugged the cloud provider's Python API until checkpoint downloads from the blob store worked well enough

   Action items

   • Run an on-call rotation for the training run, with runbooks and tooling for the most common issues, and invite more people to join it

   Lessons

   At this scale cloud hardware failures are certain, and the team concluded that automating the handling of hardware failures is necessary. Without spare nodes and burn-in tests, each failure costs hours and a replacement can bring the same fault back. The team also concluded that cloud infrastructure instability is something to always prepare for, because the customer has no control over the underlying hardware. A checkpoint large enough to need hundreds of multi-gigabyte downloads makes the storage path part of recovery time.

   View case
5. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke

   FREE NOW2019

   Symptoms

   At 16:00 CEST a developer reported in the #ask-platform Slack channel that a deployment to the Prelive cluster was failing, and a second report followed at 16:06.

   Pods PendingScale-up failure

   Trigger

   Roughly two to three days earlier, the CentOS repository mirror was restructured and container-selinux-2.68-1.el7.noarch.rpm moved, which made the static download URL hardcoded in kops nodeup invalid.

   Root cause

   Dependency failure

   Component

   Node operating system imageNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   Developers asked for help in the #ask-platform Slack channel because their deployments were failing.

   Remediation

   • downloaded the package by hand on the Prelive nodes so their bootstrap could finish
   • changed the cluster AMI from Amazon Linux 2 to Debian and applied it to all clusters

   Action items

   • Create an alert that shows when nodes cannot join the cluster, possibly from information the cluster autoscaler already has
   • Document the external dependencies of node startup and look for ways to reduce them
   • Change the AMI back to Amazon Linux 2 Minimal once kops fixes the issue
   • Document node troubleshooting better: what happens during bootstrap and where the logs are

   Lessons

   Applying cluster changes through kops and Jenkins worked well, and troubleshooting quickly showed that the failing deployments came from missing cluster capacity. What went wrong was that nobody noticed, over a long period, that some autoscaling groups held more worker instances than Kubernetes had nodes. The team was lucky that no large number of production spot instances was terminated during the problem and that it was noticed during office hours.

   View case

By root cause1