Cases3 cases

1 to 3 of 3

1. IL-0478 A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training steps

   Amazon2026

   Symptoms

   In Amazon's large-scale foundation model training, training steps took about 0.3 seconds longer (8.7 s instead of 8.4 s) when a node had a GPU whose network adapter had failed. No hardware failure alert fired, and standard node checks such as GPU burn-in and short NCCL tests rarely catch such nodes.

   Throughput degradation

   Trigger

   The network adapter of one GPU on a node failed.

   Root cause

   Hardware failure

   Component

   NCCLNIC, cable and switch port link

   Contributing factor

   Monitoring blind spot

   Detection

   No hardware alert or standard health check showed the fault. It showed as slower training steps, in the per-node network device status and transmission rate that Guard monitors, and as elevated latency or reduced bandwidth in a 2-node sweep.

   Remediation

   • Resolved the degraded communication path, which brought training step time from 8.7 s back to 8.4 s

   Action items

   • Monitor each node's network device status continuously to find and isolate nodes whose traffic NCCL rerouted
   • Remove a node flagged by online monitoring from the healthy node pool and schedule it for offline verification
   • Run a multi-node sweep that stresses cross-node collective communication to expose degraded links and misrouted traffic
   • Trigger node sweeps after anomalies found by online monitoring and after repairs, before a node re-enters production

   Lessons

   NCCL's transparent rerouting keeps both the job and NCCL tests working while it hides a failed adapter, so checks that verify functional correctness do not find the node. Amazon relies on performance-aware monitoring during training, with step time as the main signal, and on offline verification under realistic load before nodes return to service.

   View case
2. IL-0586 Four NIC-to-T0 links flapped in rapid succession and cut a 50K-GPU pretraining job's throughput about 25% for a minute

   OpenAI2026

   Symptoms

   During a 50K GPU production pretraining job at OpenAI on Cluster A, all four links of one optical transceiver on a T0 switch flapped in rapid succession, and the job's throughput dropped by about 25% for the minute of flaps. The four links went to NICs on four different nodes, three of which were running in the job.

   Link flappingThroughput degradation

   Trigger

   An optical transceiver on a T0 switch had a glitch.

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   –

   Detection

   The NICs detected their links going down and the T0 switch recorded the same links as down; the event is shown in the job's throughput and per-node packet loss graphs.

   Remediation

   • MRC recovered the packets lost during the flaps quickly, so the impact on the job stayed relatively small

   Lessons

   With MRC, failures of T0-to-T1 links and T1 switches have almost no effect on training, but NIC-to-T0 links and T0 switches still have a measurable effect and need more care. Most NIC-to-T0 link losses are transient, and a job can often recover to full speed quickly without evicting nodes.

   View case
3. IL-0418 Machine, ICI link and optical circuit switch failures interrupted training jobs on Google's TPU v4 supercomputers

   Google2024

   Symptoms

   On Google's TPU v4 supercomputers, a failed TPU machine, inter-chip interconnect (ICI) link or optical circuit switch (OCS) stopped the training jobs that used it, because a job's TPU processes all have to be up to update weights together over ICI collectives. Hardware outages hit about 1% of training jobs; jobs on a failed machine or ICI link were evicted, rescheduled onto healthy cubes and resumed from their latest checkpoint.

   Training job failureThroughput degradation

   Root cause

   Hardware failure

   Component

   TPUNIC, cable and switch port link

   Contributing factor

   –

   Detection

   healthd on each machine monitors ICI links, PCIe channels and TPU chips against modelled values and thresholds, a PCIe MSI-X interrupt reaches libtpunet when a link goes down, a TPU errors or the driver panics and libtpunet notifies Borg, a preflight check runs before every job, and the Pod Manager queries OCS health over RPC.

   Remediation

   • Marked the affected TPU machines unavailable and evicted running jobs with notice
   • Notified affected jobs so they could write an up-to-date checkpoint, and resumed rescheduled jobs from the latest checkpoint
   • Reconfigured jobs through the optical circuit switches to use spare healthy cubes after machine and ICI link outages
   • Routed traffic around links broken by an unavailable OCS with fault-tolerant wild-first routing
   • Drained broken ICI links and repaired them online while the TPUs stayed in use
   • Sent faulty hardware to a repair workflow and returned it to the pool after automated QA
   • Prioritised recovery time for OCS components over other components
   • Upgraded OCS parts across the fleet in a planned upgrade to improve reliability

   Action items

   • Add randomized routing to ICI switches for better load balancing in the presence of faults
   • Provision a hot-standby cube on an outage and migrate accelerator state to it without writing a persistent checkpoint

   Lessons

   In a static pod every resource in a contiguous block must be healthy at once, which becomes less likely as jobs grow, so job availability falls with size. Reconfiguring at the granularity of 16-machine cubes keeps the blast radius of a failure small while letting jobs use any healthy cubes, and fault-tolerant routing keeps cubes usable through rare OCS outages at a small cost in step time.

   View case

By root cause1