Incident Library

Organisation

Meta

Meta · meta

1 case · 2024

Root cause Hardware failure
Component GPU hardware and memoryNCCLPyTorch
Contributing factor Single point of failure
Symptom GPU unavailableNCCL timeoutTraining job failure

Cases

  1. IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024

    Root cause Hardware failure