Cases2 cases

1 to 2 of 2

1. IL-0476 PCIe downgrade on one machine slowed a 128-machine V100 training job for 40 minutes through a PFC surge

   ByteDance2025

   Symptoms

   A distributed training task on 128 machines in ByteDance's production environment slowed down severely for 40 minutes but kept running, holding thousands of V100 GPUs at reduced speed. The alerting in place notified engineers only when a task stopped completely, so the slowdown itself raised no alert.

   Throughput degradation

   Trigger

   The PCIe link of one machine degraded from 6.4Gbps to 4Gbps during the training task.

   Root cause

   Hardware failure

   Component

   PCIe linkInfiniBand fabric

   Contributing factor

   No alert on limit

   Detection

   No alert fired because alerts came only when a task stopped. Engineers from several teams found the faulty machine by manual diagnosis, which took 40 minutes.

   Action items

   • Detect faulty machines automatically during training by comparing each machine's per-second monitoring metrics with the other machines (Minder)

   Lessons

   Some faults slow a task without stopping it, and alerting that waits for a halted task misses them. Logs alone are incomplete because they do not contain monitoring metrics such as PFC packet rates, and no single metric reveals every fault type, so ByteDance compares machines with each other on several metrics and treats a difference that lasts for minutes as a fault.

   View case
2. IL-0420 PCIe links on IBM's Vela GPU nodes dropped to a lower generation's speed and slowed training jobs by up to 2x

   IBM2025

   Symptoms

   On IBM's Vela AI cluster, GPU nodes with PCI-E Gen3 and Gen4 links sometimes performed like a much older PCI-E generation, for example at 4 GB/s, which resembles PCI-E Gen 1. Training jobs on those nodes slowed by anywhere from a few percent to 2x, and PCI-E link downgrades were the most frequently observed failure on Vela.

   Throughput degradation

   Root cause

   unknown

   Component

   PCIe link

   Contributing factor

   –

   Detection

   Autopilot measures PCI-E bandwidth between host and device on every node with periodic lightweight tests, and an alert fires when the average of 12 hourly samples falls below a threshold for the PCI-E generation; affected nodes are listed on a Grafana dashboard.

   Remediation

   • Reset the PCI-E device, usually by rebooting the VM, which resolved the downgrade at least 95% of the time
   • Rebooted VMs periodically

   Action items

   • Run periodic lightweight PCI-E link tests through Autopilot on every node and alert when the 12-hour average falls below a threshold
   • Keep investigating the root cause of the PCI-E link degradations

   Lessons

   In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed. Lightweight periodic tests averaged over a long window can find degraded PCI-E links accurately while customer jobs keep running, which lets site reliability engineers act before users report a problem.

   View case

By root cause2