Cases1 case

1 to 1 of 1

1. IL-0420 PCIe links on IBM's Vela GPU nodes dropped to a lower generation's speed and slowed training jobs by up to 2x

   IBM2025

   Symptoms

   On IBM's Vela AI cluster, GPU nodes with PCI-E Gen3 and Gen4 links sometimes performed like a much older PCI-E generation, for example at 4 GB/s, which resembles PCI-E Gen 1. Training jobs on those nodes slowed by anywhere from a few percent to 2x, and PCI-E link downgrades were the most frequently observed failure on Vela.

   Throughput degradation

   Root cause

   unknown

   Component

   PCIe link

   Contributing factor

   –

   Detection

   Autopilot measures PCI-E bandwidth between host and device on every node with periodic lightweight tests, and an alert fires when the average of 12 hourly samples falls below a threshold for the PCI-E generation; affected nodes are listed on a Grafana dashboard.

   Remediation

   • Reset the PCI-E device, usually by rebooting the VM, which resolved the downgrade at least 95% of the time
   • Rebooted VMs periodically

   Action items

   • Run periodic lightweight PCI-E link tests through Autopilot on every node and alert when the 12-hour average falls below a threshold
   • Keep investigating the root cause of the PCI-E link degradations

   Lessons

   In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed. Lightweight periodic tests averaged over a long window can find degraded PCI-E links accurately while customer jobs keep running, which lets site reliability engineers act before users report a problem.

   View case

By root cause1