Cases1 case

1 to 1 of 1

1. IL-0419 A Granite-20B training job on IBM's Vela ran almost 3x slower because one node's power card fault throttled its GPUs

   IBM2025

   Symptoms

   A Granite-20B training job running on 768 GPUs on IBM's Vela AI cluster suddenly showed a step time almost 3 times longer than before.

   Throughput degradation

   Trigger

   A power card issue in one of the job's 96 nodes made the node throttle its GPUs from about 400W to roughly 150W.

   Root cause

   Hardware failure

   Component

   Server power supply and power capping

   Contributing factor

   –

   Detection

   The team noticed the sudden rise in step time and debugged all 96 nodes of the job to find the node with the power card issue.

   Remediation

   • Debugged all 96 nodes of the job and found the node with the power card issue

   Action items

   • Add alerts at the infrastructure layer based on host VM logs, application logs and host BMC logs to find problem nodes quickly
   • Flag nodes with an active power break slowdown from nvidia-smi GPU counters in Autopilot and the Grafana dashboard

   Lessons

   In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed and the job restarts on healthy nodes. Alerts and health checks that locate such nodes let site reliability engineers tell users about infrastructure problems before the users notice them, and cut the time to find the root cause by over 2x.

   View case

By root cause1