Cases8 cases

3 to 7 of 8

1. IL-0586 Four NIC-to-T0 links flapped in rapid succession and cut a 50K-GPU pretraining job's throughput about 25% for a minute

   OpenAI2026

   Symptoms

   During a 50K GPU production pretraining job at OpenAI on Cluster A, all four links of one optical transceiver on a T0 switch flapped in rapid succession, and the job's throughput dropped by about 25% for the minute of flaps. The four links went to NICs on four different nodes, three of which were running in the job.

   Link flappingThroughput degradation

   Trigger

   An optical transceiver on a T0 switch had a glitch.

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   –

   Detection

   The NICs detected their links going down and the T0 switch recorded the same links as down; the event is shown in the job's throughput and per-node packet loss graphs.

   Remediation

   • MRC recovered the packets lost during the flaps quickly, so the impact on the job stayed relatively small

   Lessons

   With MRC, failures of T0-to-T1 links and T1 switches have almost no effect on training, but NIC-to-T0 links and T0 switches still have a measurable effect and need more care. Most NIC-to-T0 link losses are transient, and a job can often recover to full speed quickly without evicting nodes.

   View case
2. IL-0476 PCIe downgrade on one machine slowed a 128-machine V100 training job for 40 minutes through a PFC surge

   ByteDance2025

   Symptoms

   A distributed training task on 128 machines in ByteDance's production environment slowed down severely for 40 minutes but kept running, holding thousands of V100 GPUs at reduced speed. The alerting in place notified engineers only when a task stopped completely, so the slowdown itself raised no alert.

   Throughput degradation

   Trigger

   The PCIe link of one machine degraded from 6.4Gbps to 4Gbps during the training task.

   Root cause

   Hardware failure

   Component

   PCIe linkInfiniBand fabric

   Contributing factor

   No alert on limit

   Detection

   No alert fired because alerts came only when a task stopped. Engineers from several teams found the faulty machine by manual diagnosis, which took 40 minutes.

   Action items

   • Detect faulty machines automatically during training by comparing each machine's per-second monitoring metrics with the other machines (Minder)

   Lessons

   Some faults slow a task without stopping it, and alerting that waits for a halted task misses them. Logs alone are incomplete because they do not contain monitoring metrics such as PFC packet rates, and no single metric reveals every fault type, so ByteDance compares machines with each other on several metrics and treats a difference that lasts for minutes as a fault.

   View case
3. IL-0419 A Granite-20B training job on IBM's Vela ran almost 3x slower because one node's power card fault throttled its GPUs

   IBM2025

   Symptoms

   A Granite-20B training job running on 768 GPUs on IBM's Vela AI cluster suddenly showed a step time almost 3 times longer than before.

   Throughput degradation

   Trigger

   A power card issue in one of the job's 96 nodes made the node throttle its GPUs from about 400W to roughly 150W.

   Root cause

   Hardware failure

   Component

   Server power supply and power capping

   Contributing factor

   –

   Detection

   The team noticed the sudden rise in step time and debugged all 96 nodes of the job to find the node with the power card issue.

   Remediation

   • Debugged all 96 nodes of the job and found the node with the power card issue

   Action items

   • Add alerts at the infrastructure layer based on host VM logs, application logs and host BMC logs to find problem nodes quickly
   • Flag nodes with an active power break slowdown from nvidia-smi GPU counters in Autopilot and the Grafana dashboard

   Lessons

   In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed and the job restarts on healthy nodes. Alerts and health checks that locate such nodes let site reliability engineers tell users about infrastructure problems before the users notice them, and cut the time to find the root cause by over 2x.

   View case
4. IL-0418 Machine, ICI link and optical circuit switch failures interrupted training jobs on Google's TPU v4 supercomputers

   Google2024

   Symptoms

   On Google's TPU v4 supercomputers, a failed TPU machine, inter-chip interconnect (ICI) link or optical circuit switch (OCS) stopped the training jobs that used it, because a job's TPU processes all have to be up to update weights together over ICI collectives. Hardware outages hit about 1% of training jobs; jobs on a failed machine or ICI link were evicted, rescheduled onto healthy cubes and resumed from their latest checkpoint.

   Training job failureThroughput degradation

   Root cause

   Hardware failure

   Component

   TPUNIC, cable and switch port link

   Contributing factor

   –

   Detection

   healthd on each machine monitors ICI links, PCIe channels and TPU chips against modelled values and thresholds, a PCIe MSI-X interrupt reaches libtpunet when a link goes down, a TPU errors or the driver panics and libtpunet notifies Borg, a preflight check runs before every job, and the Pod Manager queries OCS health over RPC.

   Remediation

   • Marked the affected TPU machines unavailable and evicted running jobs with notice
   • Notified affected jobs so they could write an up-to-date checkpoint, and resumed rescheduled jobs from the latest checkpoint
   • Reconfigured jobs through the optical circuit switches to use spare healthy cubes after machine and ICI link outages
   • Routed traffic around links broken by an unavailable OCS with fault-tolerant wild-first routing
   • Drained broken ICI links and repaired them online while the TPUs stayed in use
   • Sent faulty hardware to a repair workflow and returned it to the pool after automated QA
   • Prioritised recovery time for OCS components over other components
   • Upgraded OCS parts across the fleet in a planned upgrade to improve reliability

   Action items

   • Add randomized routing to ICI switches for better load balancing in the presence of faults
   • Provision a hot-standby cube on an outage and migrate accelerator state to it without writing a persistent checkpoint

   Lessons

   In a static pod every resource in a contiguous block must be healthy at once, which becomes less likely as jobs grow, so job availability falls with size. Reconfiguring at the granularity of 16-machine cubes keeps the blast radius of a failure small while letting jobs use any healthy cubes, and fault-tolerant routing keeps cubes usable through rare OCS outages at a small cost in step time.

   View case
5. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts

   Linode2022

   Symptoms

   In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

   Intermittent timeoutsThroughput degradation

   Trigger

   One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

   Root cause

   Hardware failure

   Component

   Object storage

   Contributing factor

   Monitoring blind spotAutomation amplification

   Remediation

   • Removed the failed back-end infrastructure from the Object Storage cluster

   Action items

   • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
   • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly

   Lessons

   Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

   View case

By root cause1