IL-0482 NERSC · 2025-09-04 (published)ai-infra
Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a day
The paper, written with staff of the three operating centres, characterises GPU memory errors from ECC logs covering 67.77 million GPU device-hours on 10,693 NVIDIA Ampere GPUs in three production supercomputers: Delta at NCSA (849 GPUs, A40 and A100, 388 days logged every minute), Polaris at the Argonne Leadership Computing Facility (2,240 A100 GPUs, 259 days logged every 4 seconds, DBEs only) and Perlmutter at NERSC (7,604 A100 GPUs, 326 days logged every hour), with log periods between December 2022 and January 2025. The A100's HBM2 memory is protected by single-error correction double-error detection (SECDED) ECC, so a double-bit error is detected but cannot be corrected, and the GPU application is terminated.
Errors were bursty on all three systems, and DBEs exceeded 1,000 a day on 9.32% of Perlmutter days and 4.55% of Polaris days, even though fewer than 0.53% of GPUs ever had a DBE. Error rates differed by more than three orders of magnitude between systems, the standard deviation of the mean time between errors was large, and the autocorrelation of interarrival times was high within 24 hours with no periodicity. Summit's V100 GPUs, also with HBM2, had at most 138 DBEs in a day, a milder burstiness than the A100 systems. The authors note that one DBE can stop a long training job without ending its batch job, such as a SLURM job, so allocated GPUs sit idle unless monitoring and restart are prompt, and that a static checkpoint interval fits these bursts poorly.
- Severity
- –
- Scope
- A100 GPUs on Perlmutter (7,604 GPUs) and Polaris (2,240 GPUs): DBEs on 35 and 68 GPUs, each DBE event terminating the GPU application
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
On the Perlmutter and Polaris supercomputers, NVIDIA A100 GPUs reported uncorrectable double-bit memory errors (DBEs) in bursts: 9.32% of days on Perlmutter and 4.55% of days on Polaris had more than 1,000 DBEs. Each DBE event makes ECC raise an exception that terminates the GPU application, however many errors the event holds.
The errors were reported by ECC and collected with NVIDIA DCGM, which keeps aggregated counts of single-bit errors (SBEs), which ECC corrects, and DBEs for each GPU; a new error shows as an increase in the count. Polaris logged 39,837 DBEs in 44 events on 68 GPUs, Perlmutter 17,926 DBEs in 77 events on 35 GPUs, and Delta only 9 DBEs in 2 events. On 10/26/2023 alone Polaris logged 39,144 DBEs, while 66.02% of days on Polaris and 84.97% on Perlmutter had no DBE at all.
Error rates differed between the systems by over three orders of magnitude: the DBE rate of Polaris was 0.069 per GPU per day, 2555.56 times that of Delta and 8.41 times that of Perlmutter. The bursts also showed in the gaps between events: 27.27% of DBE interarrival times on Polaris and 51.56% on Perlmutter were under one day, while the longest gaps were 42 and 23 days.
On Perlmutter, 29 GPUs had both SBEs and DBEs, and for most DBE events there were SBEs on the same GPU on the same day. The GPUs with errors changed over time instead of staying the same, and no strong correlation was found with temperature, power or GPU utilization.
Symptom GPU unavailable
Root cause
Root cause Hardware failure
- Trigger
- –
Detection
DCGM collected the ECC error counts of each GPU, every 4 seconds on Polaris, every minute on Delta and every hour for Perlmutter.
Remediation
–
Action items
- Develop proactive and dynamic GPU monitoring based on predicting which GPUs will have errors
Lessons
SBEs on a GPU can indicate a coming DBE, and the GPUs with errors change over time, so watching only GPUs with a history of errors is not enough. Because errors come in bursts, a single mean time between errors misleads, and the authors suggest checkpoint intervals that adapt to observed memory errors. Hourly error counts still captured the bursts, but finer monitoring lets administrators react sooner. Error rates differ so much between systems that conclusions from one supercomputer may not hold for another.
Sources
Related cases
IL-0416Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026same root cause · shared GPU hardware and memory
IL-0415GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025same root cause · shared GPU hardware and memory
IL-0421Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025same root cause · shared GPU hardware and memory
IL-0414Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025same root cause · shared GPU hardware and memory
IL-0411Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checksMeta · 2024same root cause · shared GPU hardware and memory
IL-0238Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024same root cause · shared GPU hardware and memory
IL-0127Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024same root cause · shared GPU hardware and memory
IL-0239LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024same root cause · shared GPU hardware and memory
IL-0429GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022same root cause · shared GPU hardware and memory
IL-0427A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed tooBigScience · 2021same root cause · shared GPU hardware and memory
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0482
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0482/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.