Cases2 cases

1 to 2 of 2

1. IL-0419 A Granite-20B training job on IBM's Vela ran almost 3x slower because one node's power card fault throttled its GPUs

   IBM2025

   Symptoms

   A Granite-20B training job running on 768 GPUs on IBM's Vela AI cluster suddenly showed a step time almost 3 times longer than before.

   Throughput degradation

   Trigger

   A power card issue in one of the job's 96 nodes made the node throttle its GPUs from about 400W to roughly 150W.

   Root cause

   Hardware failure

   Component

   Server power supply and power capping

   Contributing factor

   –

   Detection

   The team noticed the sudden rise in step time and debugged all 96 nodes of the job to find the node with the power card issue.

   Remediation

   • Debugged all 96 nodes of the job and found the node with the power card issue

   Action items

   • Add alerts at the infrastructure layer based on host VM logs, application logs and host BMC logs to find problem nodes quickly
   • Flag nodes with an active power break slowdown from nvidia-smi GPU counters in Autopilot and the Grafana dashboard

   Lessons

   In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed and the job restarts on healthy nodes. Alerts and health checks that locate such nodes let site reliability engineers tell users about infrastructure problems before the users notice them, and cut the time to find the root cause by over 2x.

   View case
2. IL-0420 PCIe links on IBM's Vela GPU nodes dropped to a lower generation's speed and slowed training jobs by up to 2x

   IBM2025

   Symptoms

   On IBM's Vela AI cluster, GPU nodes with PCI-E Gen3 and Gen4 links sometimes performed like a much older PCI-E generation, for example at 4 GB/s, which resembles PCI-E Gen 1. Training jobs on those nodes slowed by anywhere from a few percent to 2x, and PCI-E link downgrades were the most frequently observed failure on Vela.

   Throughput degradation

   Root cause

   unknown

   Component

   PCIe link

   Contributing factor

   –

   Detection

   Autopilot measures PCI-E bandwidth between host and device on every node with periodic lightweight tests, and an alert fires when the average of 12 hourly samples falls below a threshold for the PCI-E generation; affected nodes are listed on a Grafana dashboard.

   Remediation

   • Reset the PCI-E device, usually by rebooting the VM, which resolved the downgrade at least 95% of the time
   • Rebooted VMs periodically

   Action items

   • Run periodic lightweight PCI-E link tests through Autopilot on every node and alert when the 12-hour average falls below a threshold
   • Keep investigating the root cause of the PCI-E link degradations

   Lessons

   In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed. Lightweight periodic tests averaged over a long window can find degraded PCI-E links accurately while customer jobs keep running, which lets site reliability engineers act before users report a problem.

   View case

By root cause2