Cases3 cases

1 to 3 of 3

1. IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobs

   Infrawaves2026

   Symptoms

   In Infrawaves' RoCEv2 GPU training clusters, LLM training jobs crashed because their communication collective collapsed after an RNIC port flapped. The Global Identifier (GID) of the RDMA network card's port, which in RoCEv2 is expected to stay the same after initialisation, had changed: with the default OS network manager, NetworkManager, the port briefly lost its IP address while it was down, and the GID number was incremented even though the original IP address came back. The paper describes the GID shift as non-deterministic.

   Link flappingTraining job failure

   Trigger

   Ports of the RDMA network cards (RNICs) on the GPU servers flapped and were briefly down.

   Root cause

   Software defect

   Component

   NIC, cable and switch port linkInfiniBand fabricNCCLHost routes and routing rules

   Contributing factor

   Unreviewed defaults

   Remediation

   • Configured the IP addresses with the Ubuntu Networking service instead of NetworkManager, so a port down no longer removed the address or changed the GID

   Lessons

   Host networking configuration is critical to the reliability of the collective communication library. In RoCEv2 a GID is expected to stay fixed after initialisation, but an IP configuration method that briefly drops the address when an RNIC port goes down can change it and bring down the training job. Changing the IP configuration method avoided the GID shift without the compatibility risks that driver-level patches carry in virtualised networks.

   View case
2. IL-0412 InfiniBand link errors and flapping links cost Meta's RSC-1 fabric as much as 50-75% of its bandwidth during bring-up

   Meta2024

   Symptoms

   During the bring-up of Meta's RSC-1 research cluster, the InfiniBand back-end fabric lost as much as 50-75% of its bandwidth. The paper describes the faulty links that degrade communication across such a fabric as links with high error rates, links flapping between up and down states, permanently down links, and links under high congestion in a shared cluster.

   Throughput degradationLink flappingTraining job failure

   Root cause

   Dependency failure

   Component

   InfiniBand fabricNIC, cable and switch port link

   Contributing factor

   –

   Detection

   Node health checks run every five minutes and count PCI or IB link errors among the high-severity checks; the bandwidth loss was observed during the RSC-1 bring-up.

   Remediation

   • Enabled Adaptive Routing on the InfiniBand fabric so traffic avoids congested and unhealthy links
   • Removed nodes with IB link errors from scheduling at once through high-severity health checks and rescheduled their jobs

   Lessons

   The network has to remove failures and route around them; without resilience mechanisms over 50% of bandwidth may be lost. Errors with several possible causes, such as NCCL timeouts, are easily blamed on a nearby component like the network, and networking faults spread errors across the stack.

   View case
3. IL-0418 Machine, ICI link and optical circuit switch failures interrupted training jobs on Google's TPU v4 supercomputers

   Google2024

   Symptoms

   On Google's TPU v4 supercomputers, a failed TPU machine, inter-chip interconnect (ICI) link or optical circuit switch (OCS) stopped the training jobs that used it, because a job's TPU processes all have to be up to update weights together over ICI collectives. Hardware outages hit about 1% of training jobs; jobs on a failed machine or ICI link were evicted, rescheduled onto healthy cubes and resumed from their latest checkpoint.

   Training job failureThroughput degradation

   Root cause

   Hardware failure

   Component

   TPUNIC, cable and switch port link

   Contributing factor

   –

   Detection

   healthd on each machine monitors ICI links, PCIe channels and TPU chips against modelled values and thresholds, a PCIe MSI-X interrupt reaches libtpunet when a link goes down, a TPU errors or the driver panics and libtpunet notifies Borg, a preflight check runs before every job, and the Pod Manager queries OCS health over RPC.

   Remediation

   • Marked the affected TPU machines unavailable and evicted running jobs with notice
   • Notified affected jobs so they could write an up-to-date checkpoint, and resumed rescheduled jobs from the latest checkpoint
   • Reconfigured jobs through the optical circuit switches to use spare healthy cubes after machine and ICI link outages
   • Routed traffic around links broken by an unavailable OCS with fault-tolerant wild-first routing
   • Drained broken ICI links and repaired them online while the TPUs stayed in use
   • Sent faulty hardware to a repair workflow and returned it to the pool after automated QA
   • Prioritised recovery time for OCS components over other components
   • Upgraded OCS parts across the fleet in a planned upgrade to improve reliability

   Action items

   • Add randomized routing to ICI switches for better load balancing in the presence of faults
   • Provision a hot-standby cube on an outage and migrate accelerator state to it without writing a persistent checkpoint

   Lessons

   In a static pod every resource in a contiguous block must be healthy at once, which becomes less likely as jobs grow, so job availability falls with size. Reconfiguring at the granularity of 16-machine cubes keeps the blast radius of a failure small while letting jobs use any healthy cubes, and fault-tolerant routing keeps cubes usable through rare OCS outages at a small cost in step time.

   View case

By root cause3