Incident Library

Organisation

BigScience

BigScience · bigscience

1 case · 2022

Root cause Hardware failure
Component NCCLPyTorchSlurm job scheduler
Contributing factor Unreviewed defaultsLimited debug access
Symptom GPU unavailableTraining job failure

Cases

  1. IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022

    Root cause Hardware failure