Incident Library

IL-0420 IBM · 2025-01-13 (published)ai-infra

PCIe links on IBM's Vela GPU nodes dropped to a lower generation's speed and slowed training jobs by up to 2x

Vela is IBM's AI supercomputer built into IBM Cloud, online since 2022 and roughly doubled a year later, with nodes of eight 80GB A100 GPUs that are provided as KVM virtual machines with the GPUs and network functions passed through over PCI-E, and run under Red Hat OpenShift. The paper, a technical report by IBM Research and IBM Infrastructure, describes failures seen over two years of training on Vela that slowed jobs from a few percent to over 3x or crashed them, and groups them into clear hardware failures, subtle hardware failures and software failures. It places PCI-E link degradation among the software failures, which it traces in general to system software such as firmware on the GPU boards, network cards and PCI-E switches and to user application code, but it states that the root cause of these degradations is still under investigation.

Resetting the PCI-E device, which is often done by rebooting the VM, resolved the downgrade in at least 95% of cases. IBM's mitigations are checks of PCI-E links, alerts based on application logs and periodic VM reboots: Autopilot, an open-source set of health-check daemons on each GPU node, runs lightweight PCI-E tests with NVIDIA's CUDA code alongside workloads, exports the results to Prometheus, and alerts on the 12-hour average, which the paper says eliminated false positives. In distributed training a slow node holds back the whole job until it is removed or fixed and the job restarts on fresh nodes, and IBM keeps about 10% of the nodes as a buffer pool so a job can always restart with the same number of GPUs. Together with the other alerts and dashboards, this let IBM's site reliability engineers move from debugging issues when users reported them to telling users about infrastructure problems before they noticed, and cut the time to root-cause these issues by over 2x.

Severity
–
Scope
Training jobs on Vela that included a node with a downgraded PCI-E link slowed by a few percent up to 2x
Users affected
IBM researchers training models on Vela
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On IBM's Vela AI cluster, GPU nodes with PCI-E Gen3 and Gen4 links sometimes performed like a much older PCI-E generation, for example at 4 GB/s, which resembles PCI-E Gen 1. Training jobs on those nodes slowed by anywhere from a few percent to 2x, and PCI-E link downgrades were the most frequently observed failure on Vela.

The degradation showed in PCI-E bandwidth measurements between host and device that IBM's Autopilot health checks run on every node, and on a Grafana dashboard of nodes with PCI-E link degradation; one example node had degraded below 3.4GB/s. To prevent false positives from the tests and customer workloads contending for the same resources, the alert fires only when the average of 12 hourly samples over 12 hours falls below a threshold set for each PCI-E generation.

In distributed training across hundreds or thousands of GPUs, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed.

Symptom Throughput degradation

Root cause

Root cause unknown

Component PCIe link
Contributing factor –
Trigger
–

Detection

Autopilot measures PCI-E bandwidth between host and device on every node with periodic lightweight tests, and an alert fires when the average of 12 hourly samples falls below a threshold for the PCI-E generation; affected nodes are listed on a Grafana dashboard.

Remediation

Action items

Lessons

In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed. Lightweight periodic tests averaged over a long window can find degraded PCI-E links accurately while customer jobs keep running, which lets site reliability engineers act before users report a problem.

Sources

References

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, PCIe link among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.