IL-0415 NCSA · 2025-12-10 (published)ai-infra
GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on them
Delta has 106 4-way and 8-way A100 40 GB nodes with 448 A100 GPUs and a second cluster of 152 4-way GH200 nodes with 608 H100 96 GB GPUs, sharing a Lustre storage cluster and scheduled with Slurm for HPC and ML users from universities nationwide. The paper analyses 2.5 years of operational data, 895 days from October 2022 to March 2025 for A100 and 146 days from October 2024 to March 2025 for H100, 11.7 million GPU hours in total. GPU errors were extracted from system logs with regular expressions for NVIDIA XID messages, identical errors from the same GPU within 5 seconds were coalesced into one, a job was counted as GPU-failed when a GPU error occurred within 20 seconds before it failed, and DCGM metrics at one-minute granularity gave GPU utilization. The A100 GPUs logged 14,821 critical errors in total, one every 1.4 hours system-wide and every 154 hours per node; 3,359 jobs failed on GPU errors out of 1,420,278 jobs submitted to the GPU nodes.
The paper groups GSP, fallen-off-the-bus, PMU SPI and MMU errors as GPU hardware errors and treats NVLink as the interconnect. On A100 the GSP was the most vulnerable hardware component because it lacks robust error detection and recovery, and the paper calls it a single point of failure, because the errors start spontaneously and have high downstream impact such as GPU hangs. Fallen-off-the-bus errors are integration errors, often from a loose connection between GPU and motherboard or a contact failure from thermal cycles. NVLink errors come from faulty GPU hardware, connectors or connectors installed improperly during system integration; NVLink retries packets after a CRC error, but the errors were still frequent, and when an NVLink error did not fail a job the paper suggests the link may not have been in use. Apart from MMU and NVLink errors, the GPU errors studied cannot be handled by the application, so the failed jobs had to start again from the beginning or from the nearest checkpoint. The paper also notes that AWS recommends disabling GSP for stability.
To recover a GPU node, operators drained it, waiting for the other jobs on it to finish without accepting new ones, rebooted it and let it pass a health check before scheduling jobs on it again; a node that failed the reset stayed marked failed until the GPU was tested and, if needed, replaced. The expected time to service a failed node was 0.88 hours for A100 nodes and 2.2 hours for H100 nodes, 5,700 node hours were lost to GPU downtime, and per-node availability was about 99.4% for A100 and 99.3% for H100, 9 to 10 minutes of downtime a day. Four A100 GPUs were replaced because they failed to boot. On the H100 GPUs only 3 GSP errors and no fallen-off-the-bus or NVLink errors were observed, which the paper attributes to the GH200 packaging of CPU and GPU in one module and to NVIDIA driver upgrades that Delta's engineers also saw improve GSP and NVLink stability.
- Severity
- –
- Scope
- A100 GPUs on Delta went into an error state and stayed unavailable until a GPU reset or node drain and reboot; jobs on them failed, 3,359 jobs failed on GPU errors overall and 5,700 node hours were lost to GPU downtime
- Users affected
- HPC and ML users from universities nationwide who run jobs on Delta
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
On Delta, the GPU system operated by NCSA at the University of Illinois Urbana-Champaign, A100 GPUs hit GPU System Processor (GSP) errors, GPU Fallen Off the Bus errors and NVLink errors that put the GPU into an error state, failed the user job scheduled on it and kept the GPU out of service until it was reset or its node was drained and rebooted. Over 895 days of A100 operation the system logs recorded 3,857 GSP errors (XID 119/120), 1,922 NVLink errors (XID 74) and 10 GPU Fallen Off the Bus errors (XID 79).
The errors appeared as NVIDIA XID messages in the node system logs. A GSP error is logged when the GSP, an onboard co-processor that takes driver work off the CPU, does not answer remote procedure calls from the GPU driver; in the job failure table it appears as "GSP RPC timeout". A GPU Fallen Off the Bus error is logged when the driver can no longer reach the GPU over the system bus. Across the A100 fleet a GSP error came on average every 6 hours (590 hours per node), an NVLink error every 11 hours (1,185 hours per node) and a fallen-off-the-bus error every 2,148 hours (227,668 hours per node).
The errors arrived without warning and then repeated. 99% of GSP errors appeared without any preceding error, and with a probability of 0.99 a GSP error was followed by the same error again or left the GPU inoperable; the remaining 15 cases led to PMU SPI communication errors and job failures. Over 99% of fallen-off-the-bus errors were followed by similar errors in close succession and ended with the GPU in an error state. NVLink errors also had no preceding errors: 42% (801) of them spread to two or more GPUs of the node, and 17% of those involved three or more GPUs; on the same GPU an NVLink error was followed by another NVLink error with a probability of 0.38 or left the GPU in an error state with a probability of 0.61.
The impact on jobs differed by error type. All 31 jobs that encountered a GSP RPC timeout failed (100.00%). Delta's site reliability engineers considered NVLink errors largely benign, but 43 of the 80 jobs that encountered one failed (53.75%), and in two incidents a critical NVLink error needed a GPU reset and interrupted the node for over 2 hours. In one example, a GSP error stalled the control functions of an A100 GPU, the job on it failed, its DCGM GPU utilization dropped quickly, and the node drain and full reboot took 23 node hours, from 09:08 AM to 08:21 AM the next day.
Root cause
Root cause Hardware failure
- Trigger
- Delta's site reliability engineers observed that GSP errors were highly correlated with heavy ML benchmarks; the paper names GSP firmware bugs or demanding workloads as their causes.
Detection
The errors were found as NVIDIA XID messages in the node system logs; Delta's site reliability engineers monitor GSP errors closely, and job failures showed in the Slurm scheduler database and as drops in DCGM GPU utilization.
Remediation
- Drained the node with a GSP error and rebooted it fully to bring the GPU back
- Reset GPUs to recover from critical NVLink errors
- Returned a rebooted node to scheduling only after it passed a health check
- Kept nodes that failed the reset marked failed until the GPU was tested and replaced if required
- Replaced four A100 GPUs that failed to boot
Action items
–
Lessons
Except for MMU and NVLink errors, the GPU errors studied are not handled by applications and fail the job nearly every time, so application-level recovery is largely ineffective and GPU hardware resilience has to improve. Error detection and recovery of the GSP, the PMU and their communication interfaces need duplication and error detection and correction so that they stop being single points of failure. In a simulation of a month-long job on all 608 H100 GPUs with a recovery time of 2.2 hours, the job would need 5% more GPUs to keep 99.9% job-level availability, and cutting recovery time from 2.2 hours to 5 minutes would bring that down to 2%.
Sources
References
Related cases
IL-0416Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026same root cause · shared GPU hardware and memory
IL-0482Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a dayNERSC · 2025same root cause · shared GPU hardware and memory
IL-0421Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025same root cause · shared GPU hardware and memory
IL-0414Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025same root cause · shared GPU hardware and memory
IL-0411Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checksMeta · 2024same root cause · shared GPU hardware and memory
IL-0238Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024same root cause · shared GPU hardware and memory
IL-0127Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024same root cause · shared GPU hardware and memory
IL-0239LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024same root cause · shared GPU hardware and memory
IL-0429GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022same root cause · shared GPU hardware and memory
IL-0427A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed tooBigScience · 2021same root cause · shared GPU hardware and memory
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0415
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0415/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.