Incident Library

IL-0418 Google · 2024-04-16 (published)ai-infranetworking

Machine, ICI link and optical circuit switch failures interrupted training jobs on Google's TPU v4 supercomputers

The paper by Google engineers describes the software that has run TPU v4 supercomputers in production since 2020, for Google Cloud and internal users, and reports two years of fleet statistics. A supercomputer has 64 cubes of 64 TPU v4 chips each, 4096 chips in total; a cube is one rack of 16 machines with 4 chips each, linked by ICI in a 4x4x4 mesh, and 6144 optical ICI links from the cubes go to 48 optical circuit switches, MEMS mirror arrays that the Pod Manager service programs to cross-connect cubes into the torus or twisted torus a job asks for. The Borg scheduler places jobs on cubes, the Pod Manager sets up the OCS cross-connects and monitors OCS health over RPC, libtpunet sets up routing for each job, and healthd monitors each machine.

In the earlier TPU v3 static pods, where a job needed one contiguous block of healthy machines, job availability fell sharply as jobs grew towards 1024 chips; a 1024-host job needed 99.9% availability from every host, while OCS reconfiguration lowers that to 99%. When a machine or ICI link fails, its machines are marked unavailable, running jobs are evicted with notice so they can write a checkpoint if possible, and Borg reschedules them onto spare healthy cubes, where they resume from the latest checkpoint; with cube reconfiguration, availability stays high through about 94%, which the paper equates with about 50 cubes or 3200 chips. Beyond that, faults between cubes lower availability, and for OCS outages libtpunet uses fault-tolerant wild-first routing, precomputed offline with an integer linear program, which lets a packet take one extra hop in another dimension to avoid the broken links. With one OCS unavailable, measured all-to-all throughput was 70.0 GB/s against 75.9 GB/s on a 4x4x4 torus (92.2%), 63.2 against 62.1 GB/s on a 4x4x8 twisted torus (101.7%) and 53.7 against 54.3 GB/s on a 4x8x8 twisted torus (98.8%). 95% of jobs opt in to fault-tolerant routing, and fewer than 2% of jobs run with it at any time, a share that follows OCS maintenance events and their recovery time.

Faulty hardware goes to a repair workflow that combines automatic diagnosis with technician input, and passes an automated QA process before rejoining the pool; ICI links are drained and repaired online while the TPUs stay usable by jobs that do not need those links. Google prioritises recovery time for OCS parts over other parts to keep the performance impact of OCS outages small, and a planned fleet-wide upgrade of OCS parts was made to improve reliability. The paper reports 99.98% system availability, with hardware outages experienced by about 1% of training jobs handled this way.

Severity
–
Scope
About 1% of TPU v4 training jobs experienced hardware outages; jobs on a failed machine or ICI link were evicted, rescheduled onto healthy cubes and resumed from checkpoints, and jobs routed around an unavailable OCS ran 0.5% to 8.6% slower per step
Users affected
Google Cloud customers and internal Google users of TPU v4 supercomputers
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On Google's TPU v4 supercomputers, a failed TPU machine, inter-chip interconnect (ICI) link or optical circuit switch (OCS) stopped the training jobs that used it, because a job's TPU processes all have to be up to update weights together over ICI collectives. Hardware outages hit about 1% of training jobs; jobs on a failed machine or ICI link were evicted, rescheduled onto healthy cubes and resumed from their latest checkpoint.

The failures were caught on the machines and in the network software. The healthd daemon on every machine watches the 24 unidirectional ICI links, the PCIe channels between TPUs and CPUs and the 4 TPU chips, and compares link connections and link quality with modelled values and thresholds; severe symptoms make healthd ask Borg to evict and reschedule the job. Inside a job, when a TPU sees an error, the link layer goes down or the driver panics, a PCIe MSI-X interrupt is raised to libtpunet, which notifies Borg to reschedule. A preflight check before each job runs a small sample workload and compares physical link metrics with golden thresholds, so substandard links are caught before the job starts.

In an average supercomputer, each day 0.08% of the TPU machines, 0.005% of the ICI cables and 0.04% of the optical circuit switches failed. A machine or ICI link failure removes the affected cube or cubes from the resource pool, but an unavailable OCS breaks one ICI link with the same position on every cube in the pod, so it can affect all cubes. Jobs that kept running around an unavailable OCS ran slower: step time grew by 0.5% to 8.6% across Google's recommendation, LLM and BERT workloads, and nearest-neighbour all-reduce traffic lost 50% of its throughput.

Symptom Training job failureThroughput degradation

Root cause

Root cause Hardware failure

Component TPUNIC, cable and switch port link
Contributing factor –
Trigger
–

Detection

healthd on each machine monitors ICI links, PCIe channels and TPU chips against modelled values and thresholds, a PCIe MSI-X interrupt reaches libtpunet when a link goes down, a TPU errors or the driver panics and libtpunet notifies Borg, a preflight check runs before every job, and the Pod Manager queries OCS health over RPC.

Remediation

Action items

Lessons

In a static pod every resource in a contiguous block must be healthy at once, which becomes less likely as jobs grow, so job availability falls with size. Reconfiguring at the granularity of 16-machine cubes keeps the blast radius of a failure small while letting jobs use any healthy cubes, and fault-tolerant routing keeps cubes usable through rare OCS outages at a small cost in step time.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, TPU, NIC, cable and switch port link among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.