IL-0419 IBM · 2025-01-13 (published)ai-infra

A Granite-20B training job on IBM's Vela ran almost 3x slower because one node's power card fault throttled its GPUs

Vela is IBM's AI supercomputer built into IBM Cloud, online since 2022 and roughly doubled a year later, with nodes of eight 80GB A100 GPUs linked by NVLink and NVSwitch, provided as virtual machines and run under Red Hat OpenShift for training and inference. Racks are rated for 20kW from each of two redundant power distribution units and each GPU server draws up to 6kW, so full power redundancy allows three servers per rack. To fit six, IBM and its partners built a power brake: when a power supply unit fails, updated server firmware throttles the server to 3.2kW within about 2 seconds, taking each GPU from roughly 400W to 150W, and because healthy PDU circuit breakers tolerate power surges of up to 5 seconds, this lets IBM overcommit the power available to each rack safely; after testing, IBM doubled Vela's GPU density in 2023 this way.

The paper, a technical report by IBM Research and IBM Infrastructure, describes failures seen over two years of training models on Vela, which slowed jobs by a few percent up to over 3x or crashed them. It groups a node's power feed failure with failed GPUs, HBM memory corruption, PCI-E link failures and network port failures as subtle hardware failures that usually do not crash the host but can slow or fail the application. In the Granite-20B case, one node with a power card issue applied this throttling and slowed a 768-GPU job almost 3 times until the node was found by debugging all 96 nodes. IBM keeps about 10% of nodes as a buffer pool so a job can always restart with the same number of GPUs, added alerts based on host VM logs, application logs and host BMC logs after these cases, and its Autopilot health tool has a check that flags nodes with an active power break from nvidia-smi counters on a Grafana dashboard without intensive tests.

Severity: –
Scope: A Granite-20B training job on 768 GPUs ran with an almost 3x longer step time
Users affected: The team running the Granite-20B training job
Region: –
Time to detect: –
Time to resolve: –

Symptoms

A Granite-20B training job running on 768 GPUs on IBM's Vela AI cluster suddenly showed a step time almost 3 times longer than before.

The team had to debug all 96 nodes of the job to find the cause. One node had a power card issue, and the node had throttled its power consumption on its own to protect its GPUs and its healthy power supply unit. Vela's power throttling takes each GPU from about 400W to roughly 150W, so that node computed about 3 times slower than the others and dragged down the performance of the whole job. IBM's Autopilot health checks identify nodes with this kind of power problem from the GPU counters that nvidia-smi reports, such as power break slowdown.

Symptom Throughput degradation

Root cause

Root cause Hardware failure

Component Server power supply and power capping

Contributing factor –

Trigger: A power card issue in one of the job's 96 nodes made the node throttle its GPUs from about 400W to roughly 150W.

Detection

The team noticed the sudden rise in step time and debugged all 96 nodes of the job to find the node with the power card issue.

Remediation

Debugged all 96 nodes of the job and found the node with the power card issue

Action items

Add alerts at the infrastructure layer based on host VM logs, application logs and host BMC logs to find problem nodes quickly
Flag nodes with an active power break slowdown from nvidia-smi GPU counters in Autopilot and the Grafana dashboard

Lessons

In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed and the job restarts on healthy nodes. Alerts and health checks that locate such nodes let site reliability engineers tell users about infrastructure problems before the users notice them, and cut the time to find the root cause by over 2x.

Sources

https://arxiv.org/abs/2407.05467 · paper · archived copy

References

https://github.com/IBM/autopilot

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Server power supply and power capping among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0419
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0419/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.