Cases8 cases

5 to 8 of 8

1. IL-0419 A Granite-20B training job on IBM's Vela ran almost 3x slower because one node's power card fault throttled its GPUs

   IBM2025

   Symptoms

   A Granite-20B training job running on 768 GPUs on IBM's Vela AI cluster suddenly showed a step time almost 3 times longer than before.

   Throughput degradation

   Trigger

   A power card issue in one of the job's 96 nodes made the node throttle its GPUs from about 400W to roughly 150W.

   Root cause

   Hardware failure

   Component

   Server power supply and power capping

   Contributing factor

   –

   Detection

   The team noticed the sudden rise in step time and debugged all 96 nodes of the job to find the node with the power card issue.

   Remediation

   • Debugged all 96 nodes of the job and found the node with the power card issue

   Action items

   • Add alerts at the infrastructure layer based on host VM logs, application logs and host BMC logs to find problem nodes quickly
   • Flag nodes with an active power break slowdown from nvidia-smi GPU counters in Autopilot and the Grafana dashboard

   Lessons

   In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed and the job restarts on healthy nodes. Alerts and health checks that locate such nodes let site reliability engineers tell users about infrastructure problems before the users notice them, and cut the time to find the root cause by over 2x.

   View case
2. IL-0418 Machine, ICI link and optical circuit switch failures interrupted training jobs on Google's TPU v4 supercomputers

   Google2024

   Symptoms

   On Google's TPU v4 supercomputers, a failed TPU machine, inter-chip interconnect (ICI) link or optical circuit switch (OCS) stopped the training jobs that used it, because a job's TPU processes all have to be up to update weights together over ICI collectives. Hardware outages hit about 1% of training jobs; jobs on a failed machine or ICI link were evicted, rescheduled onto healthy cubes and resumed from their latest checkpoint.

   Training job failureThroughput degradation

   Root cause

   Hardware failure

   Component

   TPUNIC, cable and switch port link

   Contributing factor

   –

   Detection

   healthd on each machine monitors ICI links, PCIe channels and TPU chips against modelled values and thresholds, a PCIe MSI-X interrupt reaches libtpunet when a link goes down, a TPU errors or the driver panics and libtpunet notifies Borg, a preflight check runs before every job, and the Pod Manager queries OCS health over RPC.

   Remediation

   • Marked the affected TPU machines unavailable and evicted running jobs with notice
   • Notified affected jobs so they could write an up-to-date checkpoint, and resumed rescheduled jobs from the latest checkpoint
   • Reconfigured jobs through the optical circuit switches to use spare healthy cubes after machine and ICI link outages
   • Routed traffic around links broken by an unavailable OCS with fault-tolerant wild-first routing
   • Drained broken ICI links and repaired them online while the TPUs stayed in use
   • Sent faulty hardware to a repair workflow and returned it to the pool after automated QA
   • Prioritised recovery time for OCS components over other components
   • Upgraded OCS parts across the fleet in a planned upgrade to improve reliability

   Action items

   • Add randomized routing to ICI switches for better load balancing in the presence of faults
   • Provision a hot-standby cube on an outage and migrate accelerator state to it without writing a persistent checkpoint

   Lessons

   In a static pod every resource in a contiguous block must be healthy at once, which becomes less likely as jobs grow, so job availability falls with size. Reconfiguring at the granularity of 16-machine cubes keeps the blast radius of a failure small while letting jobs use any healthy cubes, and fault-tolerant routing keeps cubes usable through rare OCS outages at a small cost in step time.

   View case
3. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts

   Linode2022

   Symptoms

   In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

   Intermittent timeoutsThroughput degradation

   Trigger

   One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

   Root cause

   Hardware failure

   Component

   Object storage

   Contributing factor

   Monitoring blind spotAutomation amplification

   Remediation

   • Removed the failed back-end infrastructure from the Object Storage cluster

   Action items

   • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
   • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly

   Lessons

   Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

   View case
4. IL-0429 GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadable

   Tsinghua University2022

   Symptoms

   The GLM-130B pretraining run on 96 nodes, started on 2022-05-06 at 04:00, was interrupted again and again by single nodes that broke down, lost connection or reported missing GPUs, and each time training had to be restarted from a saved step. On 2022-05-30 a restart got stuck in data loading because of a Lustre file system error, and the 2.3T text corpora could no longer be read.

   Training job failureNode NotReadyGPU unavailableStorage disconnected or stale mountsThroughput degradation

   Trigger

   Individual nodes of the 96-node allocation failed during training, through breakdowns, a lost connection, missing GPUs, a CPU error and an NVLink error, and on 2022-05-30 the Lustre file system that held the training corpus failed.

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryShared network filesystem

   Contributing factor

   –

   Detection

   The Lustre error was found when a restart from global_step23200 stayed stuck in data loading.

   Remediation

   • Restarted the training from the last saved step after each node failure
   • Changed the checkpoint saving interval from 500 steps to 100 steps
   • Switched the job to another node pool and restarted from step 15600
   • Removed the suspected replacement node n30076 from step 16600, which did not change the speed
   • Copied the 2.3T text corpora from a backup disk back to the Lustre file system, which took a few days

   View case

By root cause1