Cases1 case

1 to 1 of 1

1. IL-0418 Machine, ICI link and optical circuit switch failures interrupted training jobs on Google's TPU v4 supercomputers

   Google2024

   Symptoms

   On Google's TPU v4 supercomputers, a failed TPU machine, inter-chip interconnect (ICI) link or optical circuit switch (OCS) stopped the training jobs that used it, because a job's TPU processes all have to be up to update weights together over ICI collectives. Hardware outages hit about 1% of training jobs; jobs on a failed machine or ICI link were evicted, rescheduled onto healthy cubes and resumed from their latest checkpoint.

   Training job failureThroughput degradation

   Root cause

   Hardware failure

   Component

   TPUNIC, cable and switch port link

   Contributing factor

   –

   Detection

   healthd on each machine monitors ICI links, PCIe channels and TPU chips against modelled values and thresholds, a PCIe MSI-X interrupt reaches libtpunet when a link goes down, a TPU errors or the driver panics and libtpunet notifies Borg, a preflight check runs before every job, and the Pod Manager queries OCS health over RPC.

   Remediation

   • Marked the affected TPU machines unavailable and evicted running jobs with notice
   • Notified affected jobs so they could write an up-to-date checkpoint, and resumed rescheduled jobs from the latest checkpoint
   • Reconfigured jobs through the optical circuit switches to use spare healthy cubes after machine and ICI link outages
   • Routed traffic around links broken by an unavailable OCS with fault-tolerant wild-first routing
   • Drained broken ICI links and repaired them online while the TPUs stayed in use
   • Sent faulty hardware to a repair workflow and returned it to the pool after automated QA
   • Prioritised recovery time for OCS components over other components
   • Upgraded OCS parts across the fleet in a planned upgrade to improve reliability

   Action items

   • Add randomized routing to ICI switches for better load balancing in the presence of faults
   • Provision a hot-standby cube on an outage and migrate accelerator state to it without writing a persistent checkpoint

   Lessons

   In a static pod every resource in a contiguous block must be healthy at once, which becomes less likely as jobs grow, so job availability falls with size. Reconfiguring at the granularity of 16-machine cubes keeps the blast radius of a failure small while letting jobs use any healthy cubes, and fault-tolerant routing keeps cubes usable through rare OCS outages at a small cost in step time.

   View case

By root cause1