IL-0414 NCSA · 2025-03-14 (published)ai-infra
Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100
Delta is an HPC system operated by the National Center for Supercomputing Applications (NCSA) at the University of Illinois Urbana-Champaign. It has 448 A100 40 GB GPUs in 106 4- and 8-way nodes and 608 H100 96 GB GPUs in 152 4-way GH200 nodes, 1,056 GPUs in all, serving HPC and ML users from universities nationwide. The paper uses 2.5 years of GPU error data, 11.7 million GPU hours: 895 days from October 2022 to March 2025 on the A100 GPUs and 146 days from October 2024 to March 2025 on the H100 GPUs. GPU errors were extracted from the system logs as NVIDIA XID messages with regular expressions, identical errors on the same GPU within 5 seconds were merged into one, job outcomes came from the Slurm database, and a job counted as failed by a GPU error when the error came within 20 seconds before the job failed.
A double-bit error or consecutive single-bit errors at the same address are uncorrectable ECC memory errors. They trigger row remapping, which replaces the faulty row with a spare row at the next GPU reset; when no spare rows are left the GPU reports a row remapping failure. Error containment terminates the processes using the faulty address, and dynamic page offlining marks the page unusable without a reset. On H100 the per-GPU MTBE for uncorrectable memory errors was 88,768 hours against 283,271 hours on A100, 3.2× lower, and the per-GB MTBE was 8,521,728 hours against 11,330,826 hours, 24% lower. The authors attribute this mainly to the higher memory capacity (96 GB against 40 GB, 2.4×) and also suspect HBM3's lower signalling voltage, which makes bit flips more likely, and its larger number of stacks, which makes heat harder to dissipate. The recovery mechanisms eased the impact of 92% of uncorrectable memory errors on H100, but the spare rows for remapping are capped at the same 512 rows, and 8 row remapping failures appeared on H100 early in operation while none appeared on A100 over its much longer period. Two H100 GPUs were replaced for uncorrectable memory errors and row remapping failures.
Expected time to service a failed node was 2.2 hours for H100 against 0.88 hours for A100, and node availability was about 99.3% against 99.4%. In the paper's simulation, a month-long job on all 608 H100 GPUs with 2.2 hours of recovery needs 5% overprovisioning, 31 extra GPUs, to keep 99.9% availability at the job level, and 2% if recovery takes 5 minutes.
- Severity
- –
- Scope
- Delta's 608 H100 GPUs over 146 days: 24 uncorrectable ECC memory errors, failed user jobs, and 19 hours of node unavailability in one incident
- Users affected
- Delta users from universities running HPC and ML jobs
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
On NCSA's Delta system, uncorrectable ECC memory errors on the H100 GPUs ended the user jobs running on the affected GPU, and the node then needed a reboot, or a drain and GPU reset, before it was fully recovered. In the study, every job that met an H100 memory error failed.
The errors appear in the system logs as NVIDIA XID errors: XID 48 for a double-bit ECC error (DBE), XID 63 for a row remapping event (RRE), XID 64 for a row remapping failure (RRF), XID 94 for a contained memory error and XID 95 for an uncontained one. Over 146 days the H100 GPUs had 17 DBEs, 7 consecutive single-bit errors (SBEs), 24 uncorrectable ECC memory errors in all, 16 RREs, 8 RRFs, 14 contained and 19 uncontained memory errors; the consecutive SBE and uncorrectable totals are inferred from the RRE and RRF counts. The jobs that met contained ECC errors (5 of 5), DBEs (5 of 5), RRFs (3 of 3) and RREs (2 of 2) all failed.
In one incident a double-bit error (XID 48) on an H100 failed the job scheduled on that GPU, and completing the row remapping needed a node drain and reset that kept the node from accepting new jobs for 19 hours. In another, error containment stopped an uncorrectable memory error by terminating the user's job, and the node reboot that followed caused two minutes of node downtime; GPU utilisation dropped as the job ended.
Row remapping triggered by an uncorrectable error succeeded at a rate of 0.59 on H100. When it failed, the GPU still contained the error by terminating only the affected jobs and offlining the faulty page, and containment itself failed 8% of the time. Delta site reliability engineers reported that uncontained memory errors can be highly bursty and persistent, spam the console logs and leave the GPU inoperable, although the study did not observe such bursts on A100 or H100.
Root cause
Root cause Hardware failure
- Trigger
- –
Detection
The errors are logged as NVIDIA XID errors in the node system logs, and Delta site reliability engineers continuously monitor uncorrectable ECC memory errors and error-recovery failures so faulty GPUs are replaced in time.
Remediation
- Drained the node and reset the GPU to complete row remapping after a double-bit error
- Rebooted the node after a contained uncorrectable memory error
- Replaced two H100 GPUs for uncorrectable memory errors and row remapping failures
- Replaced GPUs that repeatedly emitted row remapping failures
Action items
–
Lessons
The memory error recovery of the H100 GPUs, row remapping and error containment, does not keep up with the larger memory: the spare rows stay at 512 while capacity grew 2.4×. Application-level recovery is largely ineffective against these errors, so resilience has to improve at the GPU memory and hardware level, and shorter recovery time cuts the overprovisioning large jobs need.
Sources
References
Related cases
IL-0416Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026same root cause · shared GPU hardware and memory
IL-0415GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025same root cause · shared GPU hardware and memory
IL-0482Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a dayNERSC · 2025same root cause · shared GPU hardware and memory
IL-0421Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025same root cause · shared GPU hardware and memory
IL-0411Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checksMeta · 2024same root cause · shared GPU hardware and memory
IL-0238Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024same root cause · shared GPU hardware and memory
IL-0127Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024same root cause · shared GPU hardware and memory
IL-0239LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024same root cause · shared GPU hardware and memory
IL-0429GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022same root cause · shared GPU hardware and memory
IL-0427A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed tooBigScience · 2021same root cause · shared GPU hardware and memory
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0414
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0414/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.