Incident Library

Organisation

NCSA

NCSA · ncsa

2 cases · 2025

Root cause Hardware failure2
Component GPU hardware and memory2GPU driver1
Contributing factor Single point of failure1
Symptom GPU unavailable2Training job failure2

Cases

  1. IL-0415 GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025

    Root cause Hardware failure

  2. IL-0414 Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025

    Root cause Hardware failure